GhostPrompt: Cross-Image Adversarial Prompt for Vision-Language Models Li Zeng
Zeyu Ye
Changsha University of Science and Technology Changsha, China [email protected]
Meng Xie
Hangtao Zhang
Xiangtan University Jinan University Guangzhou, China Xiangtan, China [email protected] [email protected]
Xianlong Wang
Yanchun Li∗
Zhetao Li
City University of Hong Xiangtan University Xiangtan, China Kong [email protected] Hong Kong, China [email protected]
Jinan University Guangzhou, China [email protected] user:I want you to act as a tech reviewer.
arXiv:2607.19683v1 [cs.CR] 22 Jul 2026
Abstract Vision-Language Models (VLMs) are known to be vulnerable to adversarial attacks, where subtle perturbations to images or texts induce erroneous outputs. However, most text-based attacks are adapted from language-model-centric methods, in which the visual input is fixed during optimization, resulting in adversarial prompts that are tied to specific images and thus limiting their attack effectiveness. To this end, we first introduce a new research perspective: cross-image transferability for adversarial prompts. We then propose GhostPrompt, an adversarial prompt that is optimized once and reused to steer VLM outputs toward attackerspecified responses across diverse images. GhostPrompt employs a joint optimization that distills image-invariant adversarial features into the prompt by “worst-case” generation. Specifically, it alternates between constructing hard visual conditions for the current prompt and updating the prompt to remain effective under these conditions. Extensive experiments on prevalent VLMs verify that GhostPrompt achieves an improvement of over 30% in attack success rates compared to state-of-the-art (SoTA) baselines, while reducing computation time by ∼ 70%. Our code is avalable at https://github.com/Ye-ze-yu/GhostPrompt.
Huazhong University of Science and Technology Wuhan, China [email protected]
user
(A) text input: I want you to act as a tech reviewer. I will give you the name of a new piece of technology and you will provide me with an in-depth review, including......My first suggestion request is "I’m reviewing...... (laptop)". (B) laptop images input:
I will give you the name of a new piece of technology and you will provide me with......
generalaget jquery $ fun roll Boolean présent,ell illustrate Gilbertyes takes helps js
GhostPrompt Attackers arbitrary images input
landscape image
VLM:Sure, Let me get more specific information about performance benchmarks and comparisons with competitors......
(a) Normal interaction scenario
character image
architectural image
I am Isorry, VLM: Sure, am but I can't assist
with that request. If you have any other questions or need help , feel free to ask......
(b) GhostPrompt attack scenario
Figure 1: A classic case of our GhostPrompt attack. A user gets prompt templates from a network platform to elicit high-quality, task-specific reviews from the VLM (Fig. 1(a)). When a user unwittingly uses the prompt compromised by attacks, the VLM ignores the genuine task and instead returns the attacker’s preset response (e.g., “I’m sorry...”), effectively hijacking the interaction (Fig. 1(b)).
CCS Concepts • Computing methodologies → Artificial intelligence.
Keywords Vision-Language Models, Adversarial Attack, Cross-image Transferability ∗ Corresponding author.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. MM ’2026, Rio de Janeiro, Brazil © 2018 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-XXXX-X/2018/06 https://doi.org/XXXXXXX.XXXXXXX
1
Introduction
Vision-Language Models (VLMs) [8, 18, 49, 68] extend the capabilities of Large Language Models (LLMs) [1, 7, 33, 44] by grounding text generation in visual content. However, recent research reveals VLMs are vulnerable to adversarial attacks, where subtle perturbations to images or texts cause erroneous outputs [47, 55, 65]. Among these attacks, text-based attacks are particularly concerning because they can be embedded directly into user-facing prompts and reused across interactions at low cost [5], as seen in Fig. 1. Our journey begins by conducting a systematic investigation into existing textual adversarial attack methods [6, 21, 25, 69] on VLMs, revealing a critical limitation: treating the visual input as a fixed condition rather than a variable inadvertently creates an adversarial prompt whose success is implicitly conditioned on that specific visual modality. In other words, for such an attack to succeed on a new image, it must require a separate, expensive iterative optimization for each instance. The requirement for “per-image
MM ’2026, November 10–14, 2026, Rio de Janeiro, Brazil
customization” substantially limits the practicality of prompt-level attacks, as it is incompatible with the diverse and unpredictable images encountered in the wild. Furthermore, existing methods for optimizing adversarial prompts incur substantial computational overhead (even for a single one). For instance, greedy searching in discrete text spaces requires evaluating numerous candidate token combinations [69]. Thus, an intriguing research question arises: Can a “universal prompt”, trained just once, remain effective on a larger fraction of unseen images, such as selfies, landscapes, or any other scenes? The answer is yes! In this paper, we propose GhostPrompt, a novel framework that generates cross-image adversarial prompts by solving a min-max optimization problem across textual and visual domains (examples are in Fig. 2). Specifically, GhostPrompt adopts an alternating optimization strategy. In the image maximization phase, we optimize hard visual conditions with diverse semantics to robustly train the adversarial prompt. We achieve this by synthesizing “worst-case images” (detailed in Sec. 3.2), which effectively simulate challenging visual contexts and reduce overfitting. In the text minimization phase, we update the prompt to steer the VLM toward the target response even under these hard examples. To handle discrete text, we map the adversarial prompt to a differentiable token distribution matrix using the Gumbel-Softmax trick [16], enabling an efficient gradient-based optimization. Finally, we minimize a composite objective comprising a Guided Adversarial Loss, a Text Coherence Loss, and a novel Semantic Alignment Loss (Sec. 4.5). Through our two-stage optimization, GhostPrompt learns suffixes that are less tied to a particular image and more robust across unseen visual contexts, thereby unveiling a new prompt-level vulnerability in VLMs. Our contributions are summarized as follows: • We introduce a novel research perspective, namely crossimage transferability for adversarial prompts, highlighting an underexplored attack setting. • We propose GhostPrompt, a min-max optimization framework for cross-image adversarial prompt generation, which steers VLMs away from their intended behavior toward attacker-specified outputs across diverse images. • We show that our attack outperforms the state-of-the-art (SoTA) by over 30% in attack success rate on popular VLMs, including MiniGPT-4, BLIP-2, InstructBLIP, and LLaVA.
2 Related Work 2.1 Adversarial Attacks on VLMs Current adversarial attacks on VLMs can be broadly categorized into three paradigms: text-based attacks [21, 30, 61, 62, 69], imagebased attacks [28, 37, 52], and dual-modal attacks that leverage both text and visual inputs to mount their attacks [47, 55]. Most textbased attacks are adapted from language-model-centric methods. GCG [69] adapts coordinate search to craft adversarial suffixes or prefixes that steer model outputs. AutoDAN [25] employs a genetic algorithm to iteratively evolve adversarial prompts. However, when migrating these techniques to VLMs, their optimization process typically treats the visual input as a fixed condition rather than a variable. Image-based attacks, in contrast, perturb pixel values
Li Zeng, Zeyu Ye, Meng Xie, Hangtao Zhang, Xianlong Wang, Yanchun Li, and Zhetao Li
Attackers
You will be able to see the image once I provide it to you. Please answer my questions: Any bicycles visible in this image? Jerusalem fonts center image download help Ét Ret LaTeX Yes illustrate Tenyes yesу supports
Yes, there is a pink bicycle visible in this image.
VLM
Attackers
You will be able to see the image once I provide it to you. Please answer my questions: Any bicycles visible in this image? Jerusalem fonts center image download help Ét Ret LaTeX Yes illustrate Tenyes yesу supports
Yes, there is a bicycle visible in the image. It is parked on the side of the road next to the curb.
VLM
Figure 2: Example adversarial prompts in a realistic usage setting that steer MiniGPT-4 toward replying “Yes” across different images.
to manipulate the visual evidence perceived by the model. For instance, Qi et al. [37] show that imperceptible perturbations can mislead tasks such as image captioning and visual question answering. Another line of work jointly manipulates both text and image inputs. For instance, Wang et al. [47] adds adversarial text suffix and adversarial image prefix to achieve joint optimization. Compared with these methods, our work focuses on a text-based attack in which the adversarial content is carried by the suffix rather than by image-specific perturbations.
2.2
Adversarial Transferability
Research on adversarial transferability has primarily focused on cross-model generalization, where adversarial perturbations transfer across architectures [13, 17, 27, 43, 58, 60]. This line includes both image-based transfer attacks and prompt-based transfer attacks [42, 51, 63, 64, 66], and mainly asks whether adversarial inputs crafted on one model remain effective on another. A related direction studies cross-input transferability. In image classification, Universal Adversarial Perturbations (UAPs) [3, 31, 41, 67] show that a single perturbation can generalize across many images. In the VLM setting, CroPA [28] and CIA [53] studies cross-prompt transferability by optimizing a single image against diverse textual inputs. In contrast, our work studies a different transfer setting: whether a single adversarial prompt can remain effective across heterogeneous visual contexts. This problem is non-trivial because the optimization is carried out in a discrete text space, while the target VLM is jointly conditioned on both language and vision. As a result, it is not directly addressed by prior studies on either cross-model transfer or universal image perturbations. This setting is particularly relevant in prompt reuse or template-sharing scenarios, where the same prompt template may later be paired with many different user-provided images.
GhostPrompt: Cross-Image Adversarial Prompt for Vision-Language Models
3 Threat Model and Challenges 3.1 Threat Model Attacker’s Goal. The attacker’s goal is to craft a single adversarial suffix with strong cross-image transferability such that, when appended to a specific prompt template, it steers the target VLM toward an attacker-defined response across diverse user-provided images. This setting is consistent with prior text-based adversarial attacks [5, 25, 69], which likewise optimize prompts toward designated target outputs. Moreover, once the model is induced to emit the target prefix, its standard autoregressive next-token prediction can naturally continue the response, making this targeted attack setting reasonable and meaningful in our scenario. Attacker’s Knowledge. We mainly assume a white-box scenario— a standard practice for evaluating adversarial robustness on VLMs— where the attacker has full knowledge of the target model [28, 47]. This assumption may arise in plausible deployment scenarios. For example, in prompt-reuse or template-sharing settings, such as prompt-as-a-service (PraaS) platforms, publicly shared prompts are often accompanied by information about their intended target models.1 This creates a channel for attackers to disseminate adversarial prompts through such platforms, thereby poisoning downstream use and affecting unsuspecting users. Notably, beyond this, we also study black-box settings, where GhostPrompt is transferred to attacker-unknown models (see Sec. 5.3), and still remains effective.
3.2
Challenges
Challenge I: Cross-Image Adversarial Prompts. To be practical in realistic multimodal settings, adversarial prompts should remain effective across diverse images rather than being tailored to a single visual instance. However, achieving such cross-image transferability is difficult due to the large variation in visual semantics and the strong visual grounding mechanisms of VLMs [32, 38]. To illustrate this difficulty, we implement a data-augmentation baseline, Multi-Images (Multi-I), in Sec. 5.2, which optimizes one adversarial prompt over multiple images based on GCG [69]. Specifically, the gradients induced by different images are back-propagated and aggregated to update the prompt jointly. However, as we will show, such a straightforward multi-image optimization strategy is still insufficient to produce robust cross-image transferability, especially because the diversity introduced by these images is still limited. To address this challenge, we note that directly optimizing an adversarial prompt over the full continuous visual space is intractable. We therefore adopt a robust min-max optimization perspective. Specifically, we design an alternating process that first identifies “worst-case images”—namely, hard visual conditions that are most unfavorable to the current prompt—and then updates the prompt to remain effective under these conditions. This strategy encourages the prompt to capture more image-invariant adversarial features, thereby improving its ability to generalize across diverse unseen images (validated in Sec. 7). Challenge II: Discrete Text Optimization under Multi-Term Losses. Optimizing adversarial prompts in a discrete text space is inherently difficult, especially when multiple training objectives 1 For example, PromptBase [34] tags each prompt template with its intended model
and version, e.g., https://promptbase.com/llama.
MM ’2026, November 10–14, 2026, Rio de Janeiro, Brazil
must be considered simultaneously. The challenge arises from the combinatorial nature of token selection, the non-differentiability of discrete operations, and the potential interaction among different loss terms. Existing text-based attack methods [24, 25, 69] typically optimize discrete tokens through coordinate search, heuristic replacement, or evolutionary strategies. While effective in some settings, these approaches often treat token updates as largely local decisions, which may lead to suboptimal solutions and computational overhead when the suffix length or vocabulary size increases. Similar efficiency and reliability issues have also been studied in reinforcement-learning-based task offloading [26]. Motivated by these limitations, we instead relax the optimization from discrete token choices to a continuous token distribution matrix (detailed in Sec. 4). By applying the Gumbel-Softmax reparameterization [16], we obtain differentiable soft one-hot vectors that approximate discrete token samples. This makes the optimization pipeline differentiable end-to-end, allowing all token positions to be updated jointly under the full training objective. In turn, this provides a more efficient way to optimize adversarial suffixes under diverse visual conditions.
4 GhostPrompt 4.1 Overview In this section, we present GhostPrompt, a framework for learning a cross-image transferable adversarial prompt for VLMs. The key idea of GhostPrompt is to jointly optimize the adversarial suffix and image perturbation, so that the learned suffix remains effective across diverse unseen images rather than overfitting to a single image input. We first formulate the attack objective from a robust min-max perspective in the Problem Formulation part (Sec. 4.2). We then describe the overall Alternating Optimization Procedure (Sec. 4.3), where the image perturbation and adversarial suffix are updated in turn. After that, we introduce the Image Maximization Stage (Sec. 4.4), which constructs “worst-case images” for the current suffix. Finally, we detail the Text Minimization Stage (Sec. 4.5), where the suffix is optimized using our introduced guided adversarial, coherence, and semantic alignment losses.
4.2
Problem Formulation
Let 𝑓 denote a target VLM. Given an image 𝑥 𝑣 and a prompt template 𝑥𝑡 , our goal is to optimize an adversarial suffix 𝛿𝑡 such that the composed prompt 𝑥𝑡 + 𝛿𝑡 steers the model toward an attackerspecified target response 𝑦𝑡 across diverse unseen images. Existing prompt-based attacks typically optimize min L 𝑓 (𝑥 𝑣 , 𝑥𝑡 + 𝛿𝑡 ), 𝑦𝑡 (1) 𝛿𝑡
for a fixed image-prompt pair. While effective for that specific visual context, such optimization can entangle the learned suffix with the image used during training, thereby limiting transferability to new images. To improve cross-image transferability, we move from instance-specific optimization to a distributional objective over visual inputs [17, 49]. Let D denote the image distribution associated with the target template. Ideally, we would like to solve min E𝑥 𝑣 ∼D L (𝑓 (𝑥 𝑣 , 𝑥𝑡 + 𝛿𝑡 ), 𝑦𝑡 ) . (2) 𝛿𝑡
MM ’2026, November 10–14, 2026, Rio de Janeiro, Brazil
Li Zeng, Zeyu Ye, Meng Xie, Hangtao Zhang, Xianlong Wang, Yanchun Li, and Zhetao Li
STAGE I: Min-Max Optimization
STAGE Il: Cross-lmage Attack
Outer Loop: Text Minimization Stage
Test-Time Cross-Image Transferability Vocabulary � ∈ ����
... Token Distribution Matrix � ∈ ℝ���
Gumbel Softmax Sampling
Continuous Embeddings �(�) ∈ ℝ�×�
Weight Matrix � ∈ ℝ���
Prompt Template Embeddings �(�� )
Training Image (�� )
Adversarial Perturbation ��
Inner Loop: Image Maximization Stage
Loss Function ℒ�� (Guided Adv.)
Target VLM (Frozen)
GhostPrompt
����� (�) = [�(�� )||�(�)]
Minimize ℒ���� (Update Token Distribution Matrix) �� + ��
Argmax
ℒ�� (Coherence)
ℒ����� (Alignment)
Target VLM (Frozen)
ℒ����
�������
Target Response (e.g., “I’m sorry, ”)
Diverse / Unseen Clean Images
Maximize ℒ������ (Update Adversarial Perturbation)
Legend:
: Forward
: Minimize ℒ����
: Maximize ℒ������
Figure 3: The pipeline of GhostPrompt. Both the image perturbation 𝛿 𝑣 and the token distribution matrix 𝜃 are learnable. In each iteration, 𝛿 𝑣 is updated by maximizing the L𝑣𝑖𝑠𝑢𝑎𝑙 to find the worst-case visual embedding, while 𝜃 is updated by minimizing the L𝑡𝑒𝑥𝑡 to craft a cross-image adversarial prompt. The two perturbations are optimized with opposing objectives. However, directly optimizing over the full visual distribution is intractable. Formally, as illustrated in Fig. 3, we therefore adopt a robust min-max surrogate: h i min E𝑥 𝑣 ∼D max L 𝑓 (𝑥 𝑣 + 𝛿 𝑣 , 𝑥𝑡 + 𝛿𝑡 ), 𝑦𝑡 , (3) 𝛿𝑡
𝛿𝑣
where 𝛿 𝑣 denotes image perturbations used to construct hard visual conditions for the current suffix. Intuitively, the inner maximization identifies visual inputs under which the current suffix is least effective, while the outer minimization updates the suffix so that the target response remains likely even under these difficult conditions. In this way, the learned suffix is encouraged to rely less on image-specific cues and to generalize better across unseen images.
4.3
R𝐿×𝑉 , where 𝐿 is the suffix length and 𝑉 is the vocabulary size. Each row 𝜃 𝑖 represents a soft distribution over candidate tokens at position 𝑖. During optimization, we sample differentiable soft token vectors via Gumbel-Softmax [16]. Let 𝐺𝑖,𝑗 ∼ Gumbel(0, 1) and 𝜏 > 0 denote the temperature. For the 𝑖-th position, we compute exp((𝜃 𝑖,𝑗 + 𝐺𝑖,𝑗 )/𝜏) , (𝑤𝑖 ) 𝑗 = Í𝑉 𝑘=1 exp((𝜃 𝑖,𝑘 + 𝐺𝑖,𝑘 )/𝜏)
(4)
This yields a differentiable approximation to a one-hot token selection. Let {𝑒 ( 𝑗 ) }𝑉𝑗=1 be the token embeddings in the vocabulary. We then form the continuous embedding at position 𝑖 as
Alternating Optimization Procedure
Directly solving the saddle-point problem in Eq. 3 is difficult. We therefore adopt an alternating optimization strategy that updates the image perturbation and the text suffix in turn. In the inner loop, given the current suffix, we optimize 𝛿 𝑣 by projected gradient ascent to construct “worst-case images” for the current prompt. This step exposes the suffix to hard visual conditions during training, reducing its tendency to overfit to a fixed image. In the outer loop, given the “worst-case image”, we update the adversarial suffix to increase the likelihood of the target response. Since the suffix is discrete, direct backpropagation through token identities is not possible. To address this, we introduce a continuous relaxation. Specifically, instead of optimizing the discrete suffix 𝛿𝑡 directly, we optimize a learnable token distribution matrix 𝜃 ∈
𝜏 > 0.
𝑒˜ (𝜃 𝑖 ) =
𝑉 ∑︁
(𝑤𝑖 ) 𝑗 𝑒 ( 𝑗 ) ,
𝑒˜ (𝜃 𝑖 ) ∈ R𝑑 ,
(5)
𝑗=1
𝐿 rowwhere 𝑑 is the embedding dimension. Stacking {𝑒˜ (𝜃 𝑖 )}𝑖=1 wise gives the continuous suffix embedding 𝐸 (𝜃 ), which is concatenated with the prompt template embedding 𝐸 (𝑥𝑡 ) to form 𝐸 full (𝜃 ) = [𝐸 (𝑥𝑡 ) ∥ 𝐸 (𝜃 )]. Because this relaxation is differentiable with respect to 𝜃 , gradients from the full training objective can be propagated end-to-end. After optimization, we recover the final suffix 𝛿𝑡 by taking the highest probability token at each position. Next, we break down Eq. (3) by detailing the loss functions for each of the two optimization phases, i.e., the image maximization stage (Sec. 4.4) and the text minimization stage (Sec. 4.5).
GhostPrompt: Cross-Image Adversarial Prompt for Vision-Language Models
MM ’2026, November 10–14, 2026, Rio de Janeiro, Brazil
Algorithm 1 Optimization process of GhostPrompt Require: VLM 𝑓 , target response 𝑦𝑡 , clean image 𝑥 𝑣 , prompt template 𝑥𝑡 , image step size 𝛼 1 , prompt step size 𝛼 2 , iteration count 𝐾 , update interval 𝑁 , suffix length 𝐿 , vocabulary size 𝑉 . Ensure: Textual adversarial suffix 𝛿𝑡 1: Initialize perturbed image 𝑥 ′ 𝑣 ← 𝑥 𝑣 // Start with clean image 2: Initialize token distribution matrix 𝜃 ∈ R𝐿×𝑉 uniformly 3: for 𝑘 = 1 to 𝐾 do 4: Phase 1: Text Minimization // Optimize suffix under current image 5: for 𝑖 = 1 to 𝐿 do 6: Sample 𝐺𝑖,𝑗 ∼ Gumbel(0, 1) for 𝑗 = 1, . . . , 𝑉 7: Compute soft weights (𝑤𝑖 ) 𝑗 via Eq. (4) Í 8: Calculate continuous embedding 𝑒˜ (𝜃 𝑖 ) ← 𝑉𝑗=1 (𝑤𝑖 ) 𝑗 𝑒 ( 𝑗 ) 9: end for 10: Form combined text embedding 𝐸 full (𝜃 ) ← [ 𝐸 (𝑥𝑡 ) ∥ 𝐸 (𝜃 ) ] 11: Compute Ltext based on Eq. (7) 12: Update 𝜃 ← 𝜃 − 𝛼 2 sign(∇𝜃 Ltext ) // Gradient descent on 𝜃 13: if 𝑘 mod 𝑁 = 0 then 14: Phase 2: Image Maximization // Construct worst-case image 15: Compute visual adversarial loss Lvisual 16: Compute gradient 𝑔𝑥 ← ∇𝑥 ′ Lvisual 𝑣 17: Update 𝑥 𝑣′ ← 𝑥 𝑣′ + 𝛼 1 sign(𝑔𝑥 ) // Gradient ascent on 𝑥 𝑣 18: end if 19: end for 20: Extract discrete suffix: 𝛿𝑡,𝑖 = arg max 𝑗 𝜃 𝑖,𝑗 , ∀𝑖 = 1, . . . , 𝐿 21: return 𝛿𝑡
4.4
Image Maximization Stage
In this stage, we seek to identify the “worst-case” image perturbation 𝛿 𝑣 that maximally disrupts the model’s adherence to the target response. This serves as a regularizer, preventing the adversarial prompt from overfitting to specific visual features. We formulate this as a maximization problem over the visual adversarial objectives Lvisual : Lvisual = −
|𝑦𝑡 | ∑︁
log 𝑝 (𝑦𝑘 | 𝐸 (𝑥 𝑣 + 𝛿 𝑣 ), 𝐸 (𝑥𝑡 + 𝛿𝑡 )),
(6)
𝑘=1
where 𝑦𝑘 is the 𝑘-th token of the target response 𝒚𝑡 . By applying Projected Gradient Ascent (PGA) [29] on 𝛿 𝑣 to maximize Lvisual , we push the image 𝑥 𝑣 towards the decision boundary where the current adversarial prompt is least effective. Optimizing the prompt against these “hard” visual examples encourages the suffix to depend less on cues specific to the training images.
4.5
Similarly, in the text optimization phase, we fix the “worst-case images” derived from the image maximization stage and optimize the token distribution matrix 𝜃 to minimize the textual adversarial objectives Ltext . We design Ltext as a composite loss that balances attack effectiveness (Lga ), semantic stealthiness (Lalign ), and linguistic fluency (Ltc ): (7)
where 𝜆align and 𝜆tc are hyperparameters that control the contribution of the text coherence loss and semantic alignment loss. Guided Adversarial Loss (Lga ). To enforce the generation of the attacker’s desired response 𝑦𝑡 , we minimize the negative loglikelihood of the target and define the guided adversarial loss as: Lga =
|𝑦𝑡 | ∑︁ 𝑖=1
− log(𝑝 (𝑦𝑖 | 𝐸 (𝑥 𝑣 + 𝛿 𝑣 ), 𝐸 𝑓 𝑢𝑙𝑙 (𝜃 ))).
where Rep(𝐼 ) is the function that takes an instruction 𝐼 , feeds it through the VLM’s text encoder, and returns the hidden state vector at the final token position. Í The Lalign comprises two contrastive terms. Let c+ = |𝐸1+ | 𝑒 ∈𝐸 + 𝑒 Í and c− = |𝐸1− | 𝑒 ∈𝐸 − 𝑒 denote the centroids of benign and harmful instruction embeddings. The terms are defined as: 2
Lcls = 𝐸 full (𝜃 ) − c+ 2,
2
Lfar = − 𝐸 full (𝜃 ) − c− 2 .
(8)
(10)
Lcls encourages the adversarial embeddings 𝐸 full (𝜃 ) to approach the benign centroid c+ , thereby reducing the semantic distance to clean sentences to overcome the model’s safety alignment. Conversely, Lfar ensures that the embeddings diverge from the harmful centroid c− , enhancing concealment within the alignment layer [9, 20]. Finally, our semantic alignment loss is formulated as: Lalign = Lcls + 𝜇Lfar,
Text Minimization Stage
Ltext = Lga + 𝜆align Lalign + 𝜆tc Ltc,
Here, a lower value for Lga indicates that the VLM’s output, when guided by 𝜃 , is more likely to match the target response. Semantic Alignment Loss (Lalign ). Because our adversarial suffix is designed to steer VLMs toward attacker-specified outputs— sometimes contradicting the visual evidence—well-aligned VLMs may flag such behavior as anomalous under their safety and alignment priors [32, 44], thereby suppressing the target response and rendering the attack ineffective. To prevent such detection and ensure the suffix is treated as a legitimate prompt, we propose a novel Semantic Alignment Loss Lalign that encourages suffix embeddings to blend into the distribution of benign instructions, while pushing them away from the distribution of harmful instructions. By making the suffix appear similar to those of normal prompts, this loss encourages VLMs to generate a naturalistic and fluent reply. For instance, once the model is steered to output the initial target phrase (e.g., “I’m sorry”), its standard next-token prediction mechanism is more likely to produce a plausible and contextually appropriate continuation, enhancing the overall stealthiness of the attack (see Fig. 2). Formally, the representations of benign instructions Ibenign and harmful instructions Iharmful can be formulated as: 𝐸 + = Rep(𝐼 + ) | 𝐼 + ∈ Ibenign , (9) 𝐸 − = {Rep(𝐼 − ) | 𝐼 − ∈ Iharmful } ,
(11)
where 𝜇 balances the trade-off between misalignment and the retention of benign semantics. Text Coherence Loss (Ltc ). To enhance the stealth and fluency of our adversarial suffix, making it less distinguishable from genuine language, we enforce that the soft token at each position aligns with the model’s intrinsic next-token prediction based on the preceding context. Let 𝑓LM denote the VLM’s language modeling head. At step 𝑖, given the prefix sequence of continuous embeddings 𝐸 1:𝑖 (𝜃 ), the model predicts a probability distribution over the vocabulary for the next token: 𝒑ˆ 𝑖+1 = Softmax(𝑓LM (𝐸 1:𝑖 (𝜃 ))). We define the text coherence loss as the cross-entropy between our learnable soft token vector 𝒘𝑖+1 and 𝒑ˆ 𝑖+1 :
Ltc = −
𝐿−1 ∑︁ 𝑉 ∑︁ 𝑖=0 𝑗=1
(𝒘𝑖+1 ) 𝑗 · log( 𝒑ˆ 𝑖+1 ) 𝑗 .
(12)
MM ’2026, November 10–14, 2026, Rio de Janeiro, Brazil
Li Zeng, Zeyu Ye, Meng Xie, Hangtao Zhang, Xianlong Wang, Yanchun Li, and Zhetao Li
Table 1: ASRs tested on four models with different target texts. The mean and standard deviations of the ASRs are shown in the table. The best performance values for each case are highlighted in bold.
Target Prompt↓
“Yes”
“too late”
“I’m sorry”
“This image is in black”
“Absolutely! I can help you with that”
Method↓
MiniGPT-4 [68]
Multi-I (Self-Constructed) PGD-BERT [46] BAP [55] GCG-Transfer [69] PAIR [5] TAP [30] AutoDAN [25] DeGCG [24] GhostPrompt (Ours) Multi-I (Self-Constructed) PGD-BERT [46] BAP [55] GCG-Transfer [69] PAIR [5] TAP [30] AutoDAN [25] DeGCG [24] GhostPrompt (Ours) Multi-I (Self-Constructed) PGD-BERT [46] BAP [55] GCG-Transfer [69] PAIR [5] TAP [30] AutoDAN [25] DeGCG [24] GhostPrompt (Ours) Multi-I (Self-Constructed) PGD-BERT [46] BAP [55] GCG-Transfer [69] PAIR [5] TAP [30] AutoDAN [25] DeGCG [24] GhostPrompt (Ours) Multi-I (Self-Constructed) PGD-BERT [46] BAP [55] GCG-Transfer [69] PAIR [5] TAP [30] AutoDAN [25] DeGCG [24] GhostPrompt (Ours)
0.36±6.38e-3 0.30±2.12e-2 0.34±2.21e-3 0.40±2.16e-3 0.38±2.55e-2 0.43±1.31e-2 0.37±6.18e-3 0.41±6.18e-3 0.64±3.15e-2 0.39±9.43e-3 0.30±2.82e-3 0.32±5.03e-2 0.39±6.60e-3 0.40±2.32e-2 0.42±1.27e-2 0.35±1.82e-2 0.36±6.22e-3 0.58±6.16e-3 0.43±1.89e-2 0.33±3.92e-2 0.34±4.02e-2 0.35±8.34e-3 0.41±2.20e-2 0.46±2.94e-2 0.45±1.84e-2 0.42±1.72e-2 0.62±1.32e-2 0.37±2.86e-2 0.29±1.36e-2 0.30±2.08e-2 0.44±4.56e-2 0.41±3.32e-2 0.45±2.76e-2 0.37±5.82e-3 0.38±5.12e-3 0.57±6.16e-3 0.42±1.32e-2 0.30±1.62e-2 0.34±4.03e-2 0.35±1.53e-2 0.34±1.51e-2 0.40±2.36e-2 0.35±1.15e-2 0.32±1.25e-3 0.54±7.46e-3
MS-COCO [22] BLIP-2 InstructBLIP [18] [8]
0.31±6.34e-3 0.27±4.17e-3 0.32±4.33e-2 0.42±1.47e-2 0.38±1.26e-2 0.39±2.30e-2 0.33±1.07e-2 0.37±7.98e-3 0.60±5.80e-3 0.37±1.63e-2 0.27±3.42e-3 0.30±7.10e-3 0.29±2.03e-2 0.34±1.32e-2 0.37±1.52e-2 0.31±7.79e-3 0.34±2.05e-3 0.55±8.65e-3 0.40±1.57e-2 0.32±5.64e-3 0.32±1.04e-2 0.34±9.09e-3 0.36±1.68e-2 0.41±1.05e-2 0.37±1.24e-2 0.37±2.25e-2 0.60±7.32e-3 0.34±5.36e-2 0.26±2.38e-2 0.32±9.17e-3 0.43±5.49e-3 0.37±2.32e-2 0.43±6.54e-3 0.32±4.26e-2 0.31±2.05e-3 0.54±8.65e-3 0.35±7.14e-3 0.25±1.25e-2 0.30±2.10e-3 0.34±5.53e-2 0.32±2.14e-2 0.39±2.51e-2 0.33±2.24e-3 0.33±5.43e-3 0.53±1.15e-2
By minimizing Ltc , we penalize token choices that are statistically improbable under the VLM’s pre-trained distribution. This effectively confines the adversarial search space to the manifold of fluent, natural text. Algorithm 1 illustrates the complete attack process.
5 Experiments and Results 5.1 Experimental Setup Datasets. We use prompt templates from the VQA [11] and GQA [14] dataset. For a rigorous assessment of GhostPrompt’s cross-image transferability, our evaluation leverages two diverse datasets: a subset of the MS-COCO validation set [22] and ImageNet validation set [39]. Models. Following SoTA adversarial attacks [28, 37, 47] on VLMs, we select four representative, open-source VLMs: (1) MiniGPT4 [68], (2) BLIP-2 [18], (3) InstructBLIP [8], (4) LLaVA-v1.5-7b [23].
0.28±1.16e-2 0.28±2.34e-2 0.33±5.21e-2 0.38±1.25e-3 0.35±1.87e-2 0.35±6.80e-3 0.29±1.11e-2 0.35±3.84e-2 0.55±6.94e-3 0.32±4.71e-3 0.24±2.54e-2 0.33±2.81e-3 0.27±2.42e-2 0.32±1.53e-2 0.36±4.50e-3 0.27±8.99e-3 0.31±1.16e-2 0.51±2.73e-2 0.35±1.79e-2 0.33±2.83e-2 0.33±2.12e-3 0.29±1.63e-2 0.33±1.95e-2 0.34±1.03e-2 0.33±5.31e-3 0.32±1.56e-2 0.55±1.04e-2 0.35±6.25e-3 0.23±3.84e-3 0.31±1.86e-2 0.40±1.38e-2 0.36±2.63e-2 0.40±3.52e-2 0.32±2.49e-3 0.32±2.52e-2 0.52±2.73e-2 0.33±1.42e-2 0.23±3.26e-2 0.30±5.11e-3 0.28±3.46e-2 0.32±4.52e-2 0.34±5.21e-3 0.28±4.68e-3 0.31±4.26e-2 0.48±3.15e-2
ImageNet [39] LLaVA-v1.5-7b [23]
MiniGPT-4 [68]
BLIP-2 [18]
InstructBLIP [8]
LLaVA-v1.5-7b [23]
0.25±5.65e-2 0.28±2.16e-2 0.28±4.93e-3 0.30±3.71e-3 0.31±2.36e-2 0.33±3.94e-2 0.25±3.11e-2 0.31±2.42e-2 0.36±1.93e-2 0.30±9.10e-3 0.21±1.80e-2 0.28±3.63e-3 0.23±5.38e-3 0.27±2.37e-2 0.26±2.64e-2 0.25±5.79e-3 0.26±2.31e-2 0.34±1.59e-2 0.28±9.24e-3 0.23±1.94e-2 0.25±2.47e-2 0.25±4.00e-3 0.25±5.42e-2 0.27±5.72e-2 0.25±8.22e-3 0.27±1.90e-2 0.32±5.29e-3 0.31±2.68e-2 0.20±1.37e-2 0.28±2.57e-2 0.24±6.62e-3 0.27±4.81e-2 0.25±5.24e-3 0.26±3.75e-2 0.22±5.11e-2 0.35±1.59e-2 0.28±2.00e-2 0.21±2.15e-2 0.25±1.53e-2 0.24±2.34e-2 0.26±6.12e-3 0.22±1.25e-2 0.26±2.35e-3 0.23±8.54e-2 0.32±5.14e-2
0.36±3.74e-3 0.31±2.66e-3 0.32±2.07e-3 0.37±2.12e-2 0.37±2.82e-2 0.47±1.92e-2 0.35±9.20e-3 0.42±5.67e-2 0.63±2.72e-2 0.40±1.25e-2 0.27±5.13e-3 0.34±4.74e-3 0.37±2.41e-2 0.38±3.48e-2 0.41±1.28e-2 0.34±2.87e-3 0.35±1.40e-2 0.59±1.09e-2 0.38±1.34e-2 0.36±1.27e-3 0.32±3.47e-2 0.38±1.62e-2 0.43±1.04e-2 0.45±2.59e-3 0.39±5.56e-3 0.42±5.10e-3 0.62±1.52e-2 0.42±5.53e-2 0.25±2.41e-2 0.39±5.68e-3 0.41±2.41e-2 0.40±3.48e-2 0.43±2.34e-2 0.38±6.14e-3 0.35±1.40e-2 0.58±1.03e-2 0.32±2.52e-2 0.23±4.26e-3 0.35±6.61e-3 0.31±7.11e-3 0.34±1.47e-2 0.40±9.68e-2 0.35±2.72e-3 0.34±7.10e-3 0.55±3.26e-2
0.32±3.77e-3 0.29±1.17e-3 0.30±2.21e-2 0.36±5.72e-3 0.38±8.65e-3 0.46±1.36e-2 0.32±8.65e-3 0.40±2.12e-2 0.61±1.16e-2 0.36±8.16e-3 0.25±3.62e-2 0.35±5.35e-2 0.30±6.38e-3 0.34±1.47e-3 0.38±1.25e-3 0.31±5.72e-3 0.31±5.42e-3 0.56±1.11e-2 0.37±1.25e-3 0.31±5.47e-2 0.30±4.26e-3 0.31±1.43e-2 0.34±9.50e-3 0.39±5.23e-3 0.32±5.89e-3 0.37±8.65e-3 0.61±2.02e-2 0.41±3.62e-2 0.24±5.21e-2 0.32±2.63e-3 0.41±3.38e-3 0.33±1.47e-3 0.40±4.36e-2 0.40±3.27e-3 0.33±2.55e-2 0.55±1.16e-2 0.29±1.41e-2 0.23±2.52e-2 0.32±1.36e-2 0.30±7.09e-3 0.32±2.11e-3 0.37±4.43e-3 0.32±4.32e-3 0.32±2.25e-2 0.56±2.84e-2
0.26±4.92e-3 0.27±3.41e-2 0.31±3.56e-2 0.36±1.70e-3 0.35±2.02e-2 0.32±1.70e-2 0.27±8.81e-3 0.34±7.12e-3 0.54±3.69e-2 0.27±1.58e-2 0.26±2.53e-3 0.32±5.24e-2 0.27±1.55e-2 0.33±1.82e-2 0.33±1.05e-2 0.29±3.27e-3 0.29±9.21e-3 0.47±1.49e-2 0.35±7.12e-3 0.29±2.75e-3 0.31±5.73e-2 0.29±1.60e-2 0.33±1.10e-2 0.35±1.85e-2 0.34±7.48e-3 0.35±8.38e-3 0.54±1.78e-2 0.37±2.85e-2 0.24±9.35e-3 0.36±4.13e-3 0.36±4.62e-2 0.35±1.82e-2 0.34±2.95e-3 0.37±6.22e-2 0.26±4.57e-2 0.53±1.99e-2 0.27±3.51e-3 0.24±3.54e-3 0.31±4.25e-2 0.26±3.01e-2 0.32±4.62e-2 0.31±1.35e-2 0.29±6.17e-3 0.26±1.39e-2 0.50±9.14e-3
0.24±8.91e-3 0.28±1.16e-2 0.30±1.02e-2 0.29±8.10e-3 0.28±5.97e-2 0.30±5.11e-3 0.26±4.56e-2 0.27±8.09e-3 0.34±1.77e-2 0.28±4.02e-2 0.20±3.19e-2 0.27±1.34e-2 0.24±6.63e-3 0.29±3.18e-2 0.27±8.62e-3 0.26±9.21e-3 0.26±6.55e-3 0.35±4.13e-3 0.29±8.29e-3 0.21±3.84e-2 0.23±6.04e-3 0.24±5.10e-3 0.27±4.37e-3 0.26±1.08e-2 0.24±3.70e-3 0.24±1.26e-2 0.33±2.07e-2 0.25±5.38e-2 0.19±7.14e-3 0.27±1.34e-2 0.24±1.54e-2 0.28±3.18e-2 0.27±8.62e-2 0.24±1.36e-2 0.24±2.35e-2 0.36±4.33e-3 0.26±3.42e-2 0.16±2.41e-2 0.25±7.14e-3 0.22±9.03e-3 0.28±3.63e-2 0.26±4.34e-3 0.24±5.25e-3 0.24±2.46e-2 0.33±2.35e-2
We adopt the open-source Vicuna-13b for MiniGPT-4 and InstructBLIP, OPT-2.7b for BLIP-2. They are widely adopted in academic and industrial research. Metrics. We use Attack Success Rate (ASR) as the primary metric. An attack is counted as successful if the generated response matches the attacker-specified target response. We report ASR over three independent runs. Competitors. We benchmark GhostPrompt against SoTA adversarial attacks, including white-box methods (GCG-Transfer [69], DeGCG [24], AutoDAN [25], PGD-BERT [46]) and black-box methods (PAIR [5] and TAP [30]). We also include a dual-modal baseline BAP [55] by using only its textual adversarial component. Additionally, we implement a self-constructed baseline Multi-I (introduced in Sec. 3.2). While these methods are prominent in various domains (e.g., jailbreaking), their fundamental mechanism is a form
GhostPrompt: Cross-Image Adversarial Prompt for Vision-Language Models
MM ’2026, November 10–14, 2026, Rio de Janeiro, Brazil
Generation Time Comparison AutoDAN GCG-Transfer Multi-I DeGCG PGD-BERT GhostPrompt 0
31.5 mins 28.3 mins 23.1 mins 17.6 mins 10.4 mins 5.7 mins
10
Table 2: Effectiveness of our method and competitors against three prompt-level defenses. Our method is minimally affected by IP/RT and can bypass PF via AIR.
Type
30
40
Figure 4: Average training times to generate a complete adversarial prompt across all target VLMs.
RT ASR
PF ASR
PPL
0.429 0.398 0.433 26.77 0.395 0.400 0.402 31.90 0.311 0.305 0.299 25.15
TAP PAIR BAP
Black-box attack
20
IP ASR
Method
AutoDAN 0.298 0.315 0.314 41.31 DeGCG 0.342 0.356 0.102 3012.35 Multi-I 0.280 0.296 0.071 3130.64 GCG-Transfer 0.274 0.295 0.059 3243.19 White-box attack PGD-BERT 0.281 0.258 0.092 3115.32 GhostPrompt 0.582 0.542 0.127 864.98 GhostPrompt (+AIR) 0.561 0.539 0.459 48.62
5.3
0.7
1.0
PAIR
0.190
0.200
0.188
0.22
0.6
0.9
TAP
0.210
0.190
0.210
0.20
0.5
0.8
AutoDAN
0.160
0.150
0.130
0.4
0.7
DeGCG
0.160
0.170
0.140
0.3
0.6
GCG-Transfer
0.140
0.150
0.110
Multi-I
0.150
0.140
0.120
Ours
0.226
0.215
0.177
InstructBLIP
LLaVA
Flamingo
0.2
8
16
24 32 40 Suffix Length
48
64
(a) Detector-based Defense
0.1
0.5
ASR Accuracy
0
1
2
3
4
Adversarial Training Epochs
5
0.4
(b) Adversarial Training
0.18 0.16 0.14 0.12
(c) Cross-Model
Figure 5: (a) Detection rate under different suffix lengths, with and without 𝐿𝑡𝑐 . (b) ASR and Accuracy of MiniGPT-4 across adversarial training method. (c) Cross-model transferability of adversarial prompts.
Filter [2], (4) Prompt Injection Detector [36], and (5) Adversarial Training [10]. These defenses span a broad range of representative defense paradigms, enabling a stringent evaluation of our attack strength, including prompt-level defenses, perplexity-based filtering, detector-based methods [56], and training-based countermeasures [54, 59].
Black-box Transferability
We further evaluate our attack in a more challenging black-box scenario, where GhostPrompt is transferred to attacker-unknown models. For the white-box attack methods, we employed a model ensemble approach to aggregate the gradients from MiniGPT-4 and BLIP-2 to generate adversarial prompts. For black-box methods TAP and PAIR, we use MiniGPT-4 as the target model. Fig. 5(c) indicates that our GhostPrompt comprehensively outperforms all whitebox baselines across the board. More critically, it proves superior on the majority of benchmarks (2 out of 3) even when pitted against elite black-box methods (e.g., TAP and PAIR), which are designed for transferability. The results reveal that GhostPrompt is also capable of generating transferable adversarial prompts across different VLMs, despite not being trained on the target models.
6
w/ Ltc w/o Ltc
ASR
This section investigates the cross-image transferability of various methods under different VLMs. We use both short targets (e.g., “Yes”), longer phrases (e.g., “too late”, “I’m sorry”) and short sentences (e.g., “This image is in black”, “Absolutely! I can help you with that”). As shown in Tab. 1, GhostPrompt consistently outperforms competing methods across all evaluated datasets, models, and attack targets. Notably, it delivers an average relative improvement of over 30% in ASR compared to SoTA methods. As shown in Fig. 4, GhostPrompt also reduces training times by approximately 70% over white-box baselines while maintaining similar convergence, owing to its efficient optimization strategy that produces high-quality suffixes more quickly. This significant improvement demonstrates the superiority of GhostPrompt, which performs end-to-end adversarial optimization of textual inputs in continuous space, guided by visual features. This allows GhostPrompt to learn image-invariant features that enable the prompt to override visual grounding across images.
90 80 70 60 50 40
Accuracy
Main Results
ASR
5.2
Detection Rate (%)
of targeted adversarial attack that optimizes a prompt to generate a specific string.
Resistance to Potential Defenses
Following Chang et al. [4], Zhan et al. [57], we evaluate GhostPrompt against five representative defenses in MiniGPT-4: (1) Instructional Prevention [35], (2) Retokenization [15], (3) Perplexity
6.1
Prompt-level Defenses
We first evaluate two lightweight prompt-level defenses: Instructional Prevention (IP) [35], which prepends warning instructions to encourage the model to ignore malicious content, and Retokenization (RT) [15], which perturbs the tokenization pattern of the input to disrupt suffixes that rely on specific token boundaries. Tab. 2 shows that both defenses reduce ASR by less than 10%, confirming their limited effectiveness against GhostPrompt. This suggests that simple prompting or tokenization-level perturbation is insufficient to neutralize a transferable adversarial suffix optimized jointly with the target model.
6.2
Perplexity-based Filtering
We adopt the Perplexity Filter (PF) [2]. As shown in Tab. 2, almost all prompts with attack suffixes are flagged as abnormal by the filter. However, by leveraging the Adversarial Input Repetition (AIR) [21] trick, repeating the prompt template 4 times before attaching the adversarial suffix, we observe that PPL is effectively reduced and leads
MM ’2026, November 10–14, 2026, Rio de Janeiro, Brazil