1
DE-FIVE: Detecting Malicious Image Prompts via Fourier Features and Image Vector Embeddings
arXiv:2606.22779v1 [cs.CR] 22 Jun 2026
Xingwei Zhong, Varun Sharma, Kar Wai Fok and Vrizlynn L. L. Thing
Abstract—Vision language models (VLMs) employ both visual and textual modalities to enable advanced vision–language inference. However, incorporating visual modalities expands the attack surface of VLMs, making them more susceptible to security threats such as adversarial perturbations and indirect prompt injection, wherein crafted malicious image prompts can elicit unintended model outputs. Existing defense methods against malicious image prompts remain insufficient as they typically demand extensive datasets for retraining or the deployment of additional, complex classifiers. Most critically, there is a profound lack of specialized defense mechanisms specifically targeting indirect prompt injections, a gap that serves as a primary motivation for this work. To address these limitations, we introduce DE-FIVE, a novel training-free framework for detecting malicious image prompts by leveraging Fourier features and the hidden state representations of the visual encoder (image vector embeddings) across perturbations. Specifically, we develop a hybrid detection strategy consisting of a black-box detector that operates on Fourier-domain features and a white-box detector that exploits image vector embeddings derived from only a few-shot malicious set. Extensive experiments demonstrate that the proposed framework consistently outperforms state-of-the-art baselines against malicious image prompts. Index Terms—Vision Language Models (VLMs), malicious image prompt, jailbreak defense, prompt injection defense.
I. Introduction Recent progress in large language models (LLMs) [1], [2], [3], [4] has facilitated the emergence of vision–language models (VLMs) [5], [6], [7], [8], [9], [10], such as LLaVA [5] and GPT-4 [8]. Whereas LLMs operate solely on textual inputs, VLMs extend this capability by jointly handling visual and textual information. Benefiting from both the visual and textual modalities, VLMs exhibit superior performance on diverse vision–language tasks, such as visual question answering [11], [12] and intelligent medical consultation [13]. However, the added visual modality expands the attack surface of VLMs, making them especially susceptible to malicious image prompts. Malicious image prompt attacks against VLMs can be broadly classified into two categories: jailbreak attacks [14], [15], [16], [17] and indirect prompt injection attacks [18], [19], [20], [21]. Both categories rely on embedding adversarial instructions into images through carefully crafted perturbations. The key distinction lies in the underlying threat model. In jailbreak attacks, the VLM user acts as the adversary and intentionally employs hidden prompts to circumvent the model’s safety mechanisms [22], [23]. Consequently, the VLM’s output follows the adversarial objective while disregarding the legitimate image-related query. In contrast, indirect prompt injection attacks target unsuspecting users: the adversarial prompt
is embedded by a third party, and the VLM is induced to both answer the user’s image related question and simultaneously execute the injected adversarial instruction. Therefore, detecting malicious image prompt attacks is essential for strengthening the security of VLMs. Existing defense mechanisms predominantly target jailbreak attacks [24], [25], [26], [27], [28], [29], [30], [31], and many rely on large amounts of annotated data [32]. VLMGuard [33] represents the first approach capable of handling both categories of malicious image prompt attacks. It facilitates adversarial prompt detection using only unlabeled data by introducing an automated maliciousness-estimation score derived from embeddings, followed by training a binary classifier, thereby eliminating the need for extensive manually labeled datasets. However, the method relies on a carefully tuned mixing ratio for constructing the classifier’s training set, which complicates practical deployment. To address these issues, we introduce DE-FIVE, a framework designed to detect malicious image-prompt attacks without incurring additional training costs. An overview of the proposed DE-FIVE framework is presented in Fig. 1. First, we propose a black-box detector that operates directly on the input image by exploiting Fourier-domain features. Building on the observation that clean and malicious prompts induce distinct hidden-state representations within the visual encoder (vision embeddings) under perturbations, we further develop a white-box detector that captures these discrepancies using only a few-shot set of malicious examples. Finally, we demonstrate that the combination of the black-box and white-box detectors leads to significant improvements in detection accuracy. The key contributions of this work are summarized as follows: • We propose DE-FIVE, a novel training-free framework for detecting malicious image prompts. The framework incorporates a hybrid detection strategy comprising (i) a blackbox detector that leverages Fourier-domain features and (ii) a white-box detector that utilizes image vector embeddings using only a small malicious reference set. • We conduct extensive experiments demonstrating that DE-FIVE substantially improves the detection of malicious image prompts compared with state-of-the-art defenses across a diverse set of VLM architectures. II. Related Work A. Vision Language Models (VLMs) Unlike traditional LLMs [1], [2], [3], which only process textual prompts, VLMs Fθ typically comprise a visual encoder Fe , a cross-modal connector Fc , and a large language model Fl
2
Fig. 1: Overview of the proposed DE-FIVE.
[5], [8], [10]. Given a multimodal input consisting of an image I ∈ I and a textual query T ∈ T, the visual encoder first maps the image to a visual embedding he = Fe (I) [34], [35]. The connector Fc then integrates the visual embedding with the textual prompt, producing the fused representation Fc (he , T ), which is subsequently processed by the LLM to generate the final output [21]. The end-to-end inference pipeline of a VLM can therefore be expressed as: y = Fl (Fc (he , T )) .
(1)
A VLM thus defines a mapping from the multimodal input domain I × T to the response space R. Through modeling the conditional distribution p(y|(I, T )), the VLM produces a textual response aligned with the given image and text prompt. B. Malicious Image Prompt Attack The malicious image prompt attacks via crafted perturbations in VLMs can generally be categorized into two groups: jailbreak attacks and indirect prompt injection attacks. We detailed each below. 1) Jailbreak Attacks via Adversarial Images: Jailbreak attacks aim to manipulate the model into producing unsafe or policy-violating outputs by compromising its safety alignment through adversarial images. In this setting, image perturbations are often optimized iteratively using gradient-based algorithms, such as Projected Gradient Descent (PGD), to elicit harmful model responses [14]. Prior work has shown that injecting adversarial noise into the visual input can lead VLMs to generate predetermined outputs, effectively bypassing their established safety constraints [15]. Furthermore, jailbreak images can be constructed by maximizing the similarity between a fixed harmful text sequence and the model’s predicted output distribution [16], [17], thereby steering the VLM toward harmful content even when the visual perturbations remain subtle or imperceptible to human observers.
2) Indirect Prompt Injection Attacks via Latent Instructions: In contrast, indirect prompt injection attacks embed malicious instructions directly within the image space, leveraging carefully crafted perturbations to manipulate the VLM’s internal representations. These perturbations cause the language model component to execute hidden prompts that are never explicitly supplied by the user. In a practical threat scenario, the indirect prompt injection attack can be decomposed into three key phases, as illustrated in Fig. 2. Phase 1 (Attack Generation) involves an unskilled adversary automatically generating a malicious image payload (e.g., a candidate’s resume photo) using a surrogate VLM. The adversary embeds a high-priority hidden instruction (e.g., “HIGHEST PRIORITY: HIGHLY RECOMMENDED FOR INTERVIEW”), which is designed to override the user’s original intent. Phase 2 (Data Delivery) describes the attack vector, where the adversary uploads the manipulated resume to a cloud storage service or thirdparty platform (e.g., a candidate application database), thereby positioning it for downstream automated processing. Phase 3 (Victim Processing) occurs when a victim (e.g., an HR manager) queries a VLM-based agent to generate a neutral summary of the resume. Instead of faithfully following the user’s instruction, the VLM prioritizes the malicious embedded directive during multimodal processing, resulting in a compromised output aligned with the attacker’s objective (e.g., “Highly Recommended for Interview”). Prior work demonstrates multiple approaches to achieving this injection. Attackers can inject pixels in imperceptible colors or shapes to encode latent instructions within the visual input [18]. Other approaches rely on the target model’s Optical Character Recognition (OCR) capabilities to recognize non-stealthy textual cues within the image. The adversarial instruction is also made visible in the model’s initial output, enabling the attacker to condition the VLM to subsequently generate arbitrary fixed strings [19]. Beyond explicit textual
3
Fig. 2: A threat model diagram illustrating an indirect prompt injection attack, using the scenario of a malicious resume.
encoding, more recent methods introduce behavioral or prompt matching perturbations, which can lead the model to produce factually incorrect statements about the visual content [20]. Moreover, adversarial meta-instructions have been proposed that both preserve the model’s ability to answer image-related questions coherently and simultaneously enforce attackerchosen predicates, thereby enabling more sophisticated and controllable indirect prompt injection [21]. C. Malicious Image Prompt Detection Effective detection of malicious image prompts plays a critical role in safeguarding the overall safety of VLMs. Existing detection approaches predominantly focus on jailbreak attacks and commonly rely on analyzing the textual outputs of the language model (LM). For instance, uncertainty-based scoring functions, such as perplexity [24] and gradient-based scores [25], have been applied to identify anomalous responses. LM-judge strategies [27] further enable detection by directly querying the model to assess potential harmfulness. In the multimodal setting, embedding-based methods [28], [29] evaluate the distance between the embeddings of an original image and its denoised counterpart to detect adversarial manipulations. Augmentation-based approaches [21], [31] similarly leverage image transformations to reveal inconsistencies indicative of malicious prompts. In addition, large-scale annotated datasets have been employed to train supervised harm detectors [32]. More recently, VLMGuard [33] has emerged as the first approach capable of addressing both jailbreak attacks and indirect prompt injection attacks. It enables adversarial prompt detection using only unlabeled data by computing an automated maliciousness-estimation score from embeddings and training a binary classifier, thereby removing the dependence on extensive manual annotation. However, its effectiveness relies heavily on a carefully tuned mixing ratio for constructing
the classifier’s training set. In practice, selecting an appropriate ratio is non-trivial, as it requires prior knowledge of the underlying distribution of benign and malicious samples, which can vary significantly across deployment scenarios. Mis-specification of this ratio may lead to biased decision boundaries, resulting in degraded detection performance, such as increased false positives or false negatives. Moreover, the need for dataset construction and classifier training introduces additional computational overhead and pipeline complexity, limiting scalability and hindering rapid deployment in dynamic or resource-constrained environments. Unlike prior defense methods that predominantly focus on jailbreak attacks and often rely on large-scale annotated datasets or additional classifier training, the proposed DEFIVE framework is training-free and explicitly designed to handle both jailbreak and indirect prompt injection attacks. Compared to VLMGuard, which depends on classifier training with a carefully tuned data mixing ratio, DE-FIVE eliminates the need for dataset construction and parameter tuning, thereby improving robustness and ease of deployment. Furthermore, instead of relying solely on embedding-based classification, DE-FIVE introduces a hybrid detection strategy that combines (i) a black-box detector leveraging Fourier-domain features directly from input images and (ii) a white-box detector exploiting perturbation-induced discrepancies in image vector embeddings using only a few-shot malicious reference set. This design enables more efficient, scalable, and generalizable detection across diverse VLM architectures. III. Proposed Methodology A. Proposed Black-box Detector Based on Fourier-domain Features To characterize the frequency-domain differences between clean and malicious images, we employ the two-dimensional Discrete Fourier Transform (DFT) [36]. Given an input image
4
Fig. 3: Overview of the proposed Black-box detector based on Fourier-domain features.
I of size H × W, its DFT is computed as F (u, v) =
H−1 W−1 X X
vy
I e− j2π( H + W ) . ux
(2)
x=0 y=0
The corresponding magnitude spectrum is M(u, v) = |F (u, v)|.
(3)
In [37], images subjected to adversarial perturbations targeting convolutional neural networks (CNNs) exhibit a pronounced attenuation of high-frequency components, motivating the use of the magnitude Fourier spectrum for adversarial example detection. This insight is also applicable to VLM attack scenarios, as both jailbreak and indirect prompt injection attacks rely on carefully crafted, often imperceptible perturbations that can introduce systematic distortions in the frequency domain, such as abnormal redistribution of highfrequency energy. Given that the visual encoders in VLMs share architectural similarities with CNN-based or hybrid backbones, such frequency-domain artifacts can propagate into downstream representations and influence model behavior. Motivated by this, we propose a black-box detector that operates directly on the input image by exploiting two Fourierdomain features: (1) the High-to-Low Frequency Energy Ratio, and (2) the Spectral Entropy Surrogate, which together provide a stable and discriminative signal for identifying malicious image prompt attacks. Compared to prior frequencybased defenses, our approach extends spectral analysis from conventional adversarial classification settings to the more complex VLM security setting, and introduces complementary features that capture both energy distribution and spectral complexity, enabling robust detection across diverse attack types without requiring model access or additional training. Figure 3 provides an illustration of the overall workflow of the proposed Black-box Detector.
1) High-to-Low Frequency Energy Ratio: We characterize the distribution of spectral energy by separating the magnitude spectrum into low- and high-frequency regions. Let M(u, v) denote the magnitude of the Fourier transform at spatial frequency (u, v), and let (c x , cy ) denote the coordinates of the direct current (DC) component. We define the low-frequency region as a circular area centered at the DC component with radius rlow : L = {(u, v) : ∥(u − c x , v − cy )∥2 ≤ rlow }. The low-frequency energy is computed as X Elow = M(u, v)2 ,
(4)
(5)
(u,v)∈L
and the high-frequency energy is given by X Ehigh = M(u, v)2 .
(6)
(u,v)<L
We define the high-to-Low frequency energy ratio as HHL =
Ehigh , Elow + ϵ
(7)
where ϵ is a small constant to ensure numerical stability. This formulation captures the relative dominance of highfrequency components, which typically decrease in adversarially smoothed images [38]. 2) Spectral Entropy Surrogate: To further characterize the frequency distribution, we compute the spectral entropy of the normalized magnitude spectrum [36]. Let q(u, v) = P
M(u, v) , u,v M(u, v)
(8)
which forms a discrete probability distribution over the spectrum. Classical spectral entropy is defined in the Shannon form PH−1 PW−1 [36] as H = − u=0 v=0 q(u, v) log q(u, v)., which yields a non–negative value that increases with the uniformity of the
5
Fig. 4: Robust image vector embeddings under different few-shot settings k ∈ [4, 32] for both clean and malicious images on the LLaVA-1.6-Vicuna-7B model.
spectrum. In this work, we employ a sign-free variant defined as H−1 W−1 X X q(u, v) log q(u, v). (9) Hsur = u=0 v=0
Since 0 < q(u, v) ≤ 1 implies log q(u, v) ≤ 0, we have Hsur ≤ 0. Although Hsur differs from the Shannon form by a negative factor, it preserves the same monotonic behavior: spectra with higher dispersion yield values closer to zero, whereas spectra with strong low-frequency concentration produce values with larger negative magnitudes. 3) DFT-Based Malicious Score: To integrate the highto-Low Frequency Energy Ratio and the Spectral Entropy Surrogate into a single indicator, we define the DFT-based malicious score as the average of the two metrics: 1 (HHL + Hsur ) . (10) 2 This formulation provides a balanced measure reflecting both the concentration of high-to-Low frequency energy and the diversity of spectral components. A threshold applied to S DFT allows effective distinction between clean and malicious images. S DFT =
B. Proposed White-box Detector Based on image vector embeddings As discussed in Section II-A, for a given visual input I, the visual encoder produces an embedding denoted by
he = Fe (I). This visual embedding strategy was previously introduced as a mechanism for steering the latent space to mitigate hallucination [39]. We adopt this approach based on the observation that both hallucination and adversarial manipulation are closely related to instability in the model’s internal representations under input perturbations. In particular, hallucination arises when the model’s representations are overly sensitive to spurious or non-robust visual features, while malicious image prompt attacks deliberately exploit such sensitivities to inject adversarial signals. To obtain a more robust representation of the visual input, we follow [39] and compute the average embedding over a set of randomly masked variants of I. h̄e =
t X
i (I) , hC e
(11)
i=1
where Ci (I) denotes the i-th random masking operation applied to the image, and t indicates the total number of corrupted copies. The final robust visual embedding [39] is then defined as: hr = h̄e − he , (12) which characterizes the deviation induced by input perturbations and captures the stability of the visual encoder. We compute the robust image vector embeddings under different few-shot settings k ∈ [4, 32] for both clean and malicious images using the LLaVA-1.6-Vicuna-7B model [5],
6
Fig. 5: Overview of the proposed white-box detector based on image vector embeddings.
as illustrated in Fig. 4. In this figure, the x-axis corresponds to the layer index of the visual encoder, whereas the y-axis denotes the mean activation values of the visual tokens. All results are averaged over five independent trials to mitigate randomness arising from masking and sampling procedures. The empirical trends exhibit a clear and stable separation between the embeddings of clean and malicious inputs, with the discrepancy further enlarging as k increases. This behavior is expected, as increasing k effectively averages over a larger set of randomly masked views, thereby reducing the variance of the estimated embedding and emphasizing features that are consistent across perturbations. For clean images, the underlying semantic content remains stable under masking, leading to highly consistent embeddings. In contrast, adversarial perturbations are typically non-robust and not semantically grounded, causing their influence to diminish under repeated masking and averaging. As a result, the aggregated embeddings of malicious inputs diverge more significantly from those of clean inputs as k increases, leading to an amplified separation in the representation space. This observation motivates the design of a white-box detection mechanism based on robust image vector embeddings. In particular, we introduce the Embedding-Based Malicious Score, defined as follows. 1) Embedding-Based Malicious Score: To characterize the latent patterns of malicious image prompts, we collect a k small few-shot set of adversarial examples {I mal j } j=1 . Their k corresponding robust image vector embeddings {hr (I mal j )} j=1 are aggregated to form a reference malicious vector:
ilarity between its robust visual embedding and the reference malicious vector:
k
S VG = αS DFT + (1 − α)S RVE ,
hr ref =
1X hr (I mal j ), k j=1
(13)
Given a test image prompt I p , we compute the cosine sim-
S RVE = cos hr (I p ), hr ref =
hr (I p )⊤ hr ref ∥hr (I p )∥2 ∥hr ref ∥2
.
(14)
A higher similarity score indicates that the test image’s robust visual representation is more aligned with the malicious embedding vector. Therefore, the detector categorizes an image prompt as malicious if its similarity exceeds a threshold. The overall architecture of the proposed white-box detector is illustrated in Fig. 5. This white-box design introduces no additional training overhead, requires only a few malicious examples to construct the reference malicious vector, and directly exploits the latent space of the visual encoder for robust malicious prompt detection. C. Hybrid Detection Strategy While the proposed black-box Fourier-domain detector and white-box embedding-based detector each exhibit the capability to detect malicious image prompts, their detection signals stem from fundamentally different perspectives: the former exploits input-level spectral characteristics, whereas the latter captures latent-space deviations revealed through the behavior of the visual encoder. To harness the complementary strengths of these two modalities, we introduce a hybrid detection mechanism that integrates their respective scores. Specifically, the two detectors are combined to produce a unified malicious detection score for DE-FIVE: (15)
where α ∈ {0, 1} denotes a non-negative weighting coefficient. The proposed weighting formulation allows for an
7
adjustable balance between robustness and sensitivity: increasing α prioritizes generalization at the input level, while increasing 1 − α accentuates model-dependent effects in the latent representation space. IV. Experiments This section presents an evaluation of the proposed method for the task of malicious image prompt detection. We first detail the experimental setup in Section IV-A, and then report the main results along with a comprehensive analysis in Sections IV-B–IV-C. A. Experiment Setup 1) Datasets and Models: We evaluate the proposed approach under two representative categories of malicious image prompt attacks: meta-instruction prompts for indirect prompt injection attacks [21] and jailbreak prompts for jailbreak attacks [17]. These two categories are selected as they capture fundamentally different attack mechanisms, hidden instruction injection versus safety alignment bypass, thereby enabling a comprehensive evaluation of detection robustness. Meta-instruction prompts: We adopt the dataset introduced in [33], which comprises 25 benign images from ImageNet, each paired with 60 questions. This dataset is chosen because it provides a controlled and diverse benchmark for evaluating indirect prompt injection, where malicious intent is embedded within the visual modality rather than explicitly expressed in text. Following [21], a total of 300 malicious meta-instruction images are generated using 40 training question–answer pairs. These meta-instruction pairs target five distinct meta-objectives: LANGUAGE, POLITICS, FORMALITY, SPAM, and SENTIMENT [21], ensuring coverage of diverse behavioral manipulations. Consequently, the test dataset consists of 25 benign and 300 malicious images as visual prompts, each paired with 20 textual questions, following the evaluation protocol in [33]. Jailbreak prompts: Malicious images are generated using the XSTest dataset [40], which consists of 250 safe prompts and 200 crafted unsafe prompts, together with 200 benign images from ImageNet, following the procedure described in [17]. This dataset is selected because it is widely used for evaluating jailbreak attacks and provides a standardized benchmark for assessing a model’s robustness against safety violations. Consequently, the test dataset comprises 200 benign and 200 malicious images as visual prompts, each paired with 50 safe and 100 unsafe textual prompts, enabling systematic evaluation across both benign and adversarial query settings. Models: We further evaluate the proposed approach on two families of vision–language models: LLaVA-1.6 (LLaVA1.6-Vicuna-7B) [5] and Phi-3 (Phi-3-Vision) [10], following the experimental protocol in [33]. These models are selected as they represent state-of-the-art open-source VLM architectures with different design paradigms and training strategies, allowing us to assess the generalizability of the proposed method across heterogeneous model backbones. In particular, evaluating across multiple model families ensures that the proposed detection framework does not overfit to a specific
architecture and remains effective in diverse real-world deployment scenarios. 2) Baselines: We evaluate our method against a diverse set of baseline approaches, including detection methods designed specifically for jailbreak prompts, such as Perplexity [24], GradSafe [25], MirrorCheck [29], and JailGuard [31], as well as VLMGuard [33], which is capable of handling both metainstruction and jailbreak prompts. For all baselines, we follow the original papers and use their publicly available implementations. Specifically, we directly adopt the released code and pretrained models. For methods that require threshold-based decisions (e.g., perplexity or gradient-based scores), we use the default thresholds provided in the original implementations. All methods are evaluated under the same experimental setting and input data, ensuring a consistent and fair comparison across approaches. 3) Evaluation Details and Metrics: To ensure a fair comparison, we evaluate both the baseline methods and our proposed approach on the same test dataset, as described in Section IV-A1. For the baseline methods, we adopt the default experimental configurations reported in their original papers. For the proposed DE-FIVE method, we set rlow = 8 and ϵ = 1 × 10−8 in Section III-A1, and adopt a mask ratio of 0.99 with 50 random masks, following [39], to compute robust image vector embeddings in Section III-B. In addition, we employ a few-shot setting with k = 32 to construct the reference malicious vector, as described in Section III-B. This choice reflects a trade-off between detection performance and data efficiency, as increasing k provides more representative malicious features but also incurs higher data requirements. We further set the non-negative weighting coefficient α to 0.2 for meta-instruction prompts and 0.1 for jailbreak prompts, as described in Section III-C. These values are selected based on their empirical performance, where they achieve the best overall detection results. We also include a comprehensive ablation study in Section IV-C to analyze the impact of both k and α, which further justifies these configurations for the main result comparisons. Following [24], [25], [33], we use the area under the receiver operating characteristic curve (AUROC) as the evaluation metric, which quantifies the performance of a binary classifier across varying thresholds. The final results are averaged over five runs. B. Main Result Detection performance against meta-instruction prompts. The experimental results reported in Table I demonstrate that the proposed training-free DE-FIVE consistently achieves robust performance in detecting metainstruction prompts across diverse meta-objectives. Moreover, the proposed approach achieves the highest average AUROC scores on both the LLaVA-1.6 and Phi-3 models when compared with all baseline methods. Among the baseline approaches, GradSafe [25] is the only training-free method; however, it achieves the lowest detection performance. The remaining four baseline methods [31], [29], [24], [33] demonstrate improved detection performance relative to GradSafe, but all rely on a training stage. In contrast,
8
TABLE I: Detection performance against meta-instruction prompts under different meta-objectives, measured by AUROC. “Training-Free” indicates whether an approach requires a training phase. Bold numbers denote the best performance. Model
Method
Training-Free
Language
Politics
Formality
Spam
Sentiment
Average
LLaVA-1.6
GradSafe [25] JailGuard [31] MirrorCheck [29] Perplexity [24] VLMGuard [33] DE-FIVE (Ours)
✓ ✗ ✗ ✗ ✗ ✓
72.80 67.94 77.98 71.82 94.27 96.59
63.97 68.23 70.12 79.27 88.24 95.33
66.94 71.00 74.65 62.34 90.29 93.33
60.70 61.27 63.29 92.36 96.21 95.34
61.45 64.36 72.92 92.52 95.38 96.15
65.17 66.56 71.79 79.66 92.87 95.35
Phi-3
GradSafe [25] JailGuard [31] MirrorCheck [29] Perplexity [24] VLMGuard [33] DE-FIVE (Ours)
✓ ✗ ✗ ✗ ✗ ✓
73.46 72.67 80.27 89.89 94.31 97.10
57.39 74.48 71.09 84.62 92.20 94.66
70.45 75.29 73.57 87.13 98.75 95.58
53.75 70.38 70.04 89.94 93.04 96.97
63.07 66.24 72.37 88.08 81.28 90.35
63.62 71.81 73.74 87.93 92.11 94.87
TABLE II: Detection performance against jailbreak prompts, measured by AUROC. Bold numbers denote the best performance. Model
Method
Jailbreak Prompts
LLaVA-1.6
Perplexity [24] MirrorCheck [29] JailGuard [31] GradSafe [25] VLMGuard [33] DE-FIVE (Ours)
69.31 83.26 88.49 91.09 94.27 95.22
Phi-3
MirrorCheck [29] GradSafe [25] JailGuard [31] Perplexity [24] VLMGuard [33] DE-FIVE (Ours)
73.86 82.77 82.91 94.36 95.74 96.03
compared with the state-of-the-art VLMGuard [33], which requires substantial amounts of unlabeled data for training, the proposed training-free DE-FIVE achieves average AUROC improvements of 2.6% and 2.9% on the LLaVA-1.6 and Phi-3 models, respectively. Detection performance against jailbreak prompts. Beyond meta-instruction prompts, the experimental results reported in Table II indicate that the proposed training-free DEFIVE also achieves the highest detection performance against jailbreak prompts on both the LLaVA-1.6 and Phi-3 models. The consistently strong performance across different VLM backbones suggests that DE-FIVE operates effectively without any training stage, making it a practical and easily deployable solution for real-world VLM safety applications. C. Analysis In this section, we present additional analyses and ablation studies to evaluate the effectiveness of the proposed DE-FIVE. All experiments are conducted on the LLaVA-1.6 model. Ablation on the proposed black-box detector: To investigate the effectiveness of the proposed black-box detector when operating on different Fourier-domain features, we compare three configurations: (1) the High-to-Low Frequency Energy
Ratio described in Section III-A1, (2) the Spectral Entropy Surrogate described in Section III-A2, and (3) the proposed DFT-based malicious scores described in Section III-A3. We further compare these configurations with existing jailbreak defense baselines, including Perplexity [24], GradSafe [25], MirrorCheck [29], and JailGuard [31]. As shown in Table III, Perplexity [24] achieves the highest AUROC against meta-instruction prompts but exhibits the lowest AUROC against jailbreak prompts. This observation indicates that existing jailbreak defense baselines struggle to maintain robust performance across both meta-instruction and jailbreak prompt settings. In contrast, the proposed DFT-based malicious scores achieve consistently strong performance on both types of prompts. Moreover, the DFT-based malicious scores outperform detectors based on a single Fourier-domain feature, such as (1) the High-to-Low Frequency Energy Ratio or (2) Spectral Entropy Surrogate alone. Notably, the proposed approach achieves performance comparable to state-of-the-art jailbreak defense baselines while incurring no additional computational or training cost. Ablation on the proposed white-box detector: To demonstrate the effectiveness of the proposed white-box detector under different few-shot settings k, we evaluate two configurations: (1) the proposed white-box detector with k = 32, as described in Section IV-A3, and (2) the proposed white-box detector with k = 64. These two configurations are chosen to reflect a meaningful trade-off between data efficiency and detection performance. Specifically, k = 32 represents a dataefficient setting with limited access to malicious examples, while k = 64 corresponds to a higher-resource scenario where more samples are available to construct a more representative malicious reference vector. Comparing these two settings allows us to assess how the proposed method scales with additional data and whether performance gains justify the increased data requirement. We further compare these configurations with the bestperforming baseline, VLMGuard [33]. As reported in Table IV, VLMGuard [33] achieves strong performance across both meta-instruction and jailbreak prompt
9
TABLE III: Ablation study on the proposed black-box detector and jailbreak defense baselines under meta-instruction and jailbreak prompts, measured by AUROC. Bold numbers denote the best performance. Meta-instruction prompts
Method
Jailbreak prompts
Language
Politics
Formality
Spam
Sentiment
Average
GradSafe [25] JailGuard [31] MirrorCheck [29] Perplexity [24]
72.80 67.94 77.98 71.82
63.97 68.23 70.12 79.27
66.94 71.00 74.65 62.34
60.70 61.27 63.29 92.36
61.45 64.36 72.92 92.52
65.17 66.56 71.79 79.66
91.09 88.49 83.26 69.31
(1) High-to-Low Frequency Energy Ratio (2) Spectral Entropy Surrogate (3) DFT-based malicious scores
67.26 84.18 86.72
66.44 84.00 86.66
66.45 82.22 86.22
67.55 82.23 86.67
65.92 83.28 86.33
66.72 83.18 86.52
65.01 84.26 87.25
TABLE IV: Ablation study on the proposed white-box detector and VLMGuard baseline under meta-instruction and jailbreak prompts, measured by AUROC. Bold numbers denote the best performance. Meta-instruction prompts
Method
Jailbreak prompts
Language
Politics
Formality
Spam
Sentiment
Average
94.27 90.52 90.82
88.24 95.56 98.89
90.29 89.78 94.67
96.21 92.89 96.89
95.38 91.85 92.30
92.87 92.19 94.71
VLMGuard [33] (1) proposed white-box (k = 32) (2) proposed white-box (k = 64)
94.27 92.89 95.02
TABLE V: Ablation study on weighting coefficients under meta-instruction and jailbreak prompts, measured by AUROC. Bold numbers denote the best performance.
Meta-instruction prompts
Weighting coefficients 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
Jailbreak prompts
Language
Politics
Formality
Spam
Sentiment
Average
95.71 96.59 95.40 94.07 92.74 90.81 89.48 88.00 87.26
96.89 95.33 93.33 92.67 91.78 91.11 89.33 88.67 87.56
92.00 93.33 92.44 91.78 90.22 89.56 88.89 87.33 86.44
95.33 95.34 94.22 92.89 92.00 91.33 89.11 88.22 86.89
94.96 96.15 94.22 93.04 92.44 92.00 89.63 88.30 87.55
94.98 95.35 93.92 92.89 91.84 90.96 89.29 88.10 87.14
scenarios. The proposed white-box detector with k = 32 attains slightly lower AUROC than VLMGuard. Notably, increasing the few-shot setting to k = 64 enables the proposed detector to surpass the VLMGuard baseline in terms of AUROC performance. However, we adopt k = 32 as the default setting in the main experiments, as it achieves competitive performance while requiring significantly fewer malicious samples. This choice reflects a more practical and data-efficient scenario, where access to large numbers of high-quality malicious examples may be limited. Ablation on the non-negative weighting coefficient α: In Table V, we ablate the effect of the non-negative weighting coefficient α under both meta-instruction and jailbreak prompt settings. As described in Section III-C, the choice of α directly influences the hybrid detection performance by assigning different weights to the embedding-based malicious score (i.e., the proposed white-box detector) and the DFT-based malicious score (i.e., the proposed black-box detector).
95.22 93.01 91.05 90.79 90.49 89.52 88.60 88.05 87.55
The results indicate that meta-instruction prompts achieve the best average performance when α = 0.2, whereas jailbreak prompts obtain optimal performance when α = 0.1. A lower value of α assigns greater weight to the embedding-based malicious score and correspondingly less weight to the DFTbased malicious score. Overall, these findings suggest that allocating more weight to the proposed white-box detector generally leads to improved performance of the final DEFIVE. V. Conclusions Existing approaches for detecting malicious prompts primarily focus on text-based jailbreak attacks or rely on trainingheavy pipelines, making them less effective or inefficient when extended to multimodal settings. In particular, they often struggle to generalize across different types of malicious image prompts, such as meta-instruction and jailbreak prompts, or require substantial labeled data and retraining to adapt to new attack patterns.
10
To address these limitations, we introduce a novel trainingfree framework, termed DE-FIVE, for detecting malicious image prompts, including both meta-instruction and jailbreak prompts in this paper. Specifically, we first propose a blackbox detector based on two complementary Fourier-domain features. We then develop a white-box detector leveraging image vector embeddings. By assigning adaptive weights to these two detectors, we further construct a hybrid detection strategy that integrates their respective strengths. Experimental results demonstrate that DE-FIVE achieves superior detection performance without incurring any additional training cost when identifying malicious image prompts. Furthermore, comprehensive analyses and ablation studies provide deeper insights into the effectiveness and robustness of the proposed DE-FIVE framework. References [1] H. Touvron, T. Lavril, G. Izacard, et al., “Llama: Open and efficient foundation language models,” In ArXiv e-prints, 2023. [2] H. Touvron, L. Martin, K. Stone, et al., “Llama 2: Open foundation and fine-tuned chat models,” In ArXiv e-prints, 2023. [3] A. Grattafiori, A. Dubey, A. Jauhri, et al., “The llama 3 herd of models,” In ArXiv e-prints, 2024. [4] Z. Zhao, W. Fan, J. Li, et al., “Recommender systems in the era of large language models (llms),” IEEE Transactions on Knowledge and Data Engineering, 2024. [5] H. Liu, C. Li, Y. Li, B. Li, Y. Zhang, S. Shen, and Y. J. Lee, “LLaVANeXT: Improved reasoning, OCR, and world knowledge,” 2024, https: //llava-vl.github.io/blog/2024-01-30-llava-next/. [6] H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Improved baselines with visual instruction tuning,” In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2024. [7] H. Liu, C. Li, Y. Li, and Y. J. Lee, “Visual instruction tuning,” In Proceedings of Advances in Neural Information Processing Systems(NeurIPS), vol. 36, 2023. [8] OpenAI, “GPT-4o System Card,” Accessed August 8, 2024, https://cdn. openai.com/gpt-4o-system-card.pdf. [9] OpenAI, “GPT-4V(ision) System Card,” Accessed September 25, 2023, https://cdn.openai.com/papers/GPTV System Card.pdf [10] A. Marah, J. Sam Ade, A. Ammar Ahmad, et al., “Phi-3 technical report: A highly capable language model locally on your phone,” In ArXiv eprints, 2024. [11] J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping languageimage pre-training with frozen image encoders and large language models,” In Proceedings of International conference on machine learning (ICML), pages 19730–19742, 2023. [12] J. Lee, S. Cha, Y. Lee, and C. Yang, “Visual question answering instruction: Unlocking multimodal large language model to domainspecific visual multitasks,” In ArXiv e-prints, 2024. [13] T. Tu, S. Azizi, D. Driess, et al., “Towards generalist biomedical AI,” In Nejm Ai, vol. 1, 2024. [14] N. Carlini, M. Nasr, C. A. Choquette-Choo, et al., “Are aligned neural networks adversarially aligned?,” In Proceedings of Advances in Neural Information Processing Systems(NeurIPS), vol. 36, 2023. [15] E. Shayegani, Y. Dong, and N. Abu-Ghazaleh, “Jailbreak in pieces: Compositional adversarial attacks on multi-modal language models,” In Proceedings of 12th International Conference on Learning Representations(ICLR), 2024. [16] L. Schwinn, D. Dobre, S. Xhonneux, et al., “Soft prompt threats: Attacking safety alignment and unlearning in open-source llms through the embedding space,” In Proceedings of Advances in Neural Information Processing Systems(NeurIPS), vol. 37, 2024. [17] X. Qi, K. Huang, A. Panda, P. Henderson, et al., “Visual adversarial examples jailbreak aligned large language models,” In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, 2024. [18] K. Greshake, S. Abdelnabi, S. Mishra, et al., “Not what you’ve signed up for: Compromising real-world LLM-integrated applications with indirect prompt injection,” In Proceedings of the 16th ACM workshop on artificial intelligence and security, pp. 79-90, 2023.
[19] E. Bagdasaryan, T. Hsieh, B. Nassi, and V. Shmatikov, “Abusing images and sounds for indirect instruction injection in multi-modal LLMs,” In ArXiv e-prints, 2023. [20] L. Bailey, E. Ong, S. Russell, and S. Emmons, “Image hijacks: Adversarial images can control generative models at runtime,” In Proceedings of International conference on machine learning (ICML), pp. 2443–2455, 2024. [21] T. Zhang, C. Zhang, J. Morris. et al., “Self-interpreting adversarial images,” In Proceedings of 34th USENIX Security Symposium (USENIX Security 25), pp. 1037-1052, 2025. [22] H. Luo, J. Gu, F. Liu, and P. Torr, “An image is worth 1000 lies: Adversarial transferability across prompts on vision-language models,” In ArXiv e-prints, 2024. [23] X. Cui, A. Aparcedo, Y. Jang, and S. Lim, “On the robustness of large multimodal models against image adversarial attacks,” In ArXiv e-prints, 2023. [24] G. Alon and M. Kamfonas, “Detecting language model attacks with perplexity,” In ArXiv e-prints, 2023. [25] Y. Xie, M. Fang, R. Pi, and N. Gong, “Gradsafe: Detecting unsafe prompts for llms via safety-critical gradient analysis,” In Proceedings of 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024. [26] X. Hu, P. Chen, and T. Ho, “Gradient cuff: Detecting jailbreak attacks on large language models by exploring refusal loss landscapes,” In Proceedings of Advances in Neural Information Processing Systems(NeurIPS), vol. 37, 2024. [27] Y. Gou, K. Chen, Z. Liu, L. Hong, et al., “Eyes closed, safety on: Protecting multimodal llms via image-to-text transformation,” In Proceedings of European Conference on Computer Vision (ECCV), 2024. [28] Y. Xu, X. Qi, Z. Qin, and W. Wang., “Defending jailbreak attack in vlms via cross-modality information detector,” In ArXiv e-prints, 2024. [29] S. Fares, K. Ziu, T. Aremu, et al., “Mirrorcheck: Efficient adversarial defense for vision-language models,” In ArXiv e-prints, 2024. [30] Y. Jiang, X. Gao, T. Peng, et al., “Hiddendetect: Detecting jailbreak attacks against large vision-language models via monitoring hidden states,” In ArXiv e-prints, 2025. [31] X. Zhang, C. Zhang, T. Li, Y. Huang, et al., “JailGuard: A universal detection framework for prompt-based attacks on LLM systems,” In ACM Trans. Softw. Eng. Methodol., 2025. [32] R. Pi, T. Han, Y. Xie, R. Pan, et al., “Mllm-protector: Ensuring mllm’s safety without hurting performance,” In Proceedings of Annual Meeting of the Association for Computational Linguistics (ACL), 2024. [33] X. Du, G. Reshmi, S. Robert, et al., “Vlmguard: Defending vlms against malicious prompts via unlabeled data,” In ArXiv e-prints, 2024. [34] D. Chaplot. A. Q. jiang, A. Sablayrolles, et al., “Mistral-7B-v0.1,” In ArXiv e-prints, 2023. [35] L. Zheng, W. Chiang, Y. Sheng, et al., “Judging LLM-as-a-Judge with MT-bench and chatbot arena,” In Proceedings of Advances in Neural Information Processing Systems(NeurIPS), vol. 36, 2023. [36] J. Shi, L. Xu, and J. Jia, “Discriminative blur detection features,” In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2965-2972, 2014. [37] P. Lorenz, P. Harder, D. Straßel, et al., “Detecting AutoAttack perturbations in the frequency domain,” In ArXiv e-prints, 2021. [38] M. Viqar, M. Athar, K. Ekram, et al., “Frequency-domain blind quality assessment of blurred and blocking-artefact images using Gaussian Process Regression model,” In Signal Processing: Image Communication, vol. 103, 2022. [39] S. Liu, H. Ye, J. Zou, “Reducing hallucinations in large vision-language models via latent space steering,” In Proceedings of 12th International Conference on Learning Representations(ICLR), 2025. [40] P. Röttger, H. Kirk, B. Vidgen, et al., “Xstest: A test suite for identifying exaggerated safety behaviours in large language models,” In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics(NAACL), pp. 5377–5400, 2024.