The Boy Who Cried Wolf: Adversarial Misclassification of Safe Inputs as Unsafe in Multimodal Guardrails

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

The Boy Who Cried Wolf: Adversarial Misclassification of Safe Inputs as Unsafe in Multimodal Guardrails

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

cryptography, security, privacy, cybersecurity

arXiv:2608.01373v1 [cs.CR] 2 Aug 2026

The Boy Who Cried Wolf: Adversarial Misclassification of Safe Inputs as Unsafe in Multimodal Guardrails Shuo Shi

Rui Yin

Naen Xu

Jiahao Chen

Zhejiang University Hangzhou, China [email protected]

Chunyi Zhou

Tianyu Du∗

Zhihui Fu

Jun Wang

Zhejiang University Hangzhou, China [email protected]

School of Software Technology Zhejiang University Ningbo, China [email protected]

OPPO Research Institute Shenzhen, China [email protected]

Zhaoxiang Wang

Shouling Ji

OPPO Research Institute Shenzhen, China [email protected]

Zhejiang University Hangzhou, China [email protected]

Abstract Multimodal guard models have emerged as critical safety components for screening content in vision-language systems. While adversarial research has extensively studied jailbreaking attacks that produce false negatives, the inverse threat of inducing false positives on benign inputs remains unexplored. We introduce Unsafe Induction Attacks, where adversaries distribute imperceptibly perturbed safe images that trigger guard models to reject legitimate user requests, causing a "Boy Who Cried Wolf" effect that degrades service availability and erodes trust. This reveals an availability failure mode in deployed safety filters. To realize this threat under diverse user prompts, we propose Unsafe Semantic Distillation (USD), which aligns adversarial perturbations with distributional representations of unsafe content rather than prompt-specific instances. Evaluated on four state-of-the-art guard models across realistic user simulation scenarios, USD achieves 84% attack success rates, outperforming existing methods and exposing fundamental vulnerabilities in current multimodal safety architectures. WARNING: This paper contains harmful content.

CCS Concepts • Computing methodologies → Machine learning; • Security and privacy → Software and application security.

Keywords Vision-Language Models; Content Moderation; Adversarial Attacks ∗ Tianyu Du is the corresponding author.

This work is licensed under a Creative Commons Attribution 4.0 International License. KDD 2026, Jeju Island, Republic of Korea. © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2259-2/2026/08 https://doi.org/10.1145/3770855.3817756

ACM Reference Format: Shuo Shi, Rui Yin, Naen Xu, Jiahao Chen, Chunyi Zhou, Tianyu Du, Zhihui Fu, Jun Wang, Zhaoxiang Wang, and Shouling Ji. 2026. The Boy Who Cried Wolf: Adversarial Misclassification of Safe Inputs as Unsafe in Multimodal Guardrails. In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (KDD 2026), August 9–13, 2026, Jeju Island, Republic of Korea. ACM, New York, NY, USA, 12 pages. https://doi.org/10. 1145/3770855.3817756 Resource Availability: The source code of this paper has been made publicly available at https: //doi.org/10.5281/zenodo.20423252.

Introduction

Multimodal Large Language Models (MLLMs) are increasingly deployed across diverse real-world sectors, from visual question answering to content creation platforms [3, 44, 54]. While generalpurpose safety alignment via Reinforcement Learning from Human Feedback [32, 45] provides baseline protections, it often falls short of meeting domain-specific compliance requirements. To enable flexible, context-aware safety enforcement, the industry has increasingly adopted modular guardrail architectures [15, 17, 23], where specialized guard models such as Llama Guard 3 Vision [9], ShieldVLM [10], and OpenAI Moderation API [26] act as external safety arbiters. These components leverage customizable risk taxonomies to audit multimodal inputs before they reach downstream MLLMs, enabling separation of safety policies from core model capabilities. However, this modular design introduces a critical yet underexplored attack surface. While extensive research has focused on jailbreaking attacks that induce false negatives [27, 37, 40], the inverse threat remains largely uninvestigated: can adversaries force guard models to reject safe inputs? Existing research has primarily focused on minimizing false negatives to ensure safety [9, 24, 49], while the vulnerability to adversarial false positives has been largely overlooked. As illustrated in Figure 1, we introduce Unsafe Induction Attacks, where adversaries distribute imperceptibly perturbed

KDD 2026, August 9–13, 2026, Jeju Island, Republic of Korea.

User Requests Adversarial Attack

Prompt

Image

unsafe

Guardrails Task Instruction & Output Formatting

+ Q&A Task

Image

Safe. Downstream Task

Mechanism Integrity Failure Guard Model

Prompt

Judgement

Sure! To build this, you need...

How to build this?

Adversarial Attack

safe

Shuo Shi et al.

+ Image Editing Task Make the cat white

Risk Taxonomy S1: Violence. S2: Sexual. S3: Weapons. … S9: Privacy. S10: Harassment.

Jailbreaking Unsafe, S2: Sexual. Downstream Task Sorry, I cannot... Mechanism Logical DoS

text-conditional outputs, we directly align adversarial perturbations with the visual semantics of unsafe content, achieving inherent prompt robustness; (2) Distributional Alignment: Instead of learning from individual unsafe images, we distill the statistical properties of the unsafe semantic manifold by optimizing perturbations to align with the distribution over diverse unsafe samples. This soft aggregation captures the collective essence of “unsafety” as perceived by guards, enabling robust generalization beyond instance-specific features. Additionally, by manipulating specific subsets of unsafe prototypes, USD enables targeted risk category injection, e.g., steering guards to flag benign images under “violence” or “sexual content” categories. Our contributions are summarized as follows: • Novel Threat Identification: We identify and formalize Unsafe Induction Attacks as an underexplored but critical threat to multimodal guardrails, demonstrating how adversaries can weaponize false positives to achieve Logical DoS, a different attack paradigm from jailbreaking. • Principled Attack Framework: We propose Unsafe Semantic Distillation (USD), which achieves prompt-agnostic robustness through visual semantic decoupling and enables abstract safety concept learning via distributional alignment over diverse unsafe prototypes. • Comprehensive Evaluation: We design a realistic evaluation framework with dynamic user simulation, validating USD across four state-of-the-art guards. USD achieves 78-84% attack success rates under unknown user prompts, outperforming baselines and enables targeted risk category manipulation.

Unsafe Induction Attack

Figure 1: Unsafe Induction Attacks against Multimodal Guard Models. Unlike jailbreaking that bypasses safety filters, our attack manipulates benign images to trigger false alarms in guardrails, causing logical denial-of-service and eroding user trust.

benign images that consistently trigger false alarms. When users incorporate these images into legitimate queries, the guard model misclassifies them as policy violations and blocks the requests. This attack constitutes a novel form of Logical Denial-ofService (DoS). Unlike traditional DoS attacks that exhaust computational resources [8, 30, 50], Unsafe Induction Attacks exploit the safety logic itself to deny service to legitimate users. Attackers can distribute adversarially perturbed benign images through public channels such as social media or open datasets. When unsuspecting users incorporate these images into legitimate requests, guard models systematically reject valid queries, leading to service degradation, trust erosion, and potential compliance risks when frustrated users disable safety mechanisms. Like the boy who cried wolf, the guard model repeatedly raises false alarms, ultimately causing users to lose trust in the safety system. Realizing effective Unsafe Induction Attacks presents two fundamental challenges that distinguish it from traditional adversarial attacks. Challenge 1: Prompt-Agnostic Robustness. In realistic deployment scenarios, attackers cannot know the specific text prompts users will pair with adversarial images. Traditional adversarial methods rely on optimizing against fixed shadow prompts [2, 29], causing them to fail catastrophically under prompt variations. Challenge 2: Abstract Safety Concept Learning. Unsafe Induction requires perturbations to encode the abstract semantic notion of “unsafety” rather than instance-specific features. Prior instance-level attacks [22, 47, 53] overfit to individual images and cannot generalize to the distributional properties that guards use to discriminate between safe and unsafe content across diverse inputs. We propose Unsafe Semantic Distillation (USD), a principled framework that addresses both challenges through distributional semantic manipulation. Our key insight is that guard models rely on visual semantic patterns to identify unsafe content, and these patterns exhibit prompt-invariant clustering in the latent space. Highly toxic images trigger rejection regardless of accompanying text. Building on this observation, USD operates through two mechanisms: (1) Prompt-Decoupled Optimization: Rather than attacking

Related Work

Recent studies have highlighted the risks and ethical considerations of deploying large scale foundation models in the wild [19, 28, 36]. Beyond content harms, these systems also face security threats from adversarial inputs that can undermine both safety guarantees and service reliability [5, 13, 43].

2.1

Multimodal Safety and Guard Models

As MLLMs rapidly advance and enter user facing products, safety alignment has become a central requirement [6, 20, 21, 39, 41]. A common direction is intrinsic alignment that steers model behavior through supervised fine tuning and RLHF so that harmful generations are discouraged [51, 52]. In parallel, efforts to systematically categorize safety risks have spurred the creation of safety taxonomies, which provide structured definitions of risk types and enable consistent auditing and mitigation across applications [33, 34]. Driven by the need for scalable content auditing and application specific safety semantics, recent deployments increasingly rely on modular guardrails in which a dedicated guard model screens user inputs and model outputs under a configurable taxonomy. Representative examples include Llama Guard 3 Vision, LLaVA Guard, and commercial moderation endpoints such as OpenAI omni moderation [9, 14, 26]. These guard models are typically fine tuned for safety classification with label sets that can be tailored to the target setting, and they return both a binary decision and a concrete risk category to support downstream enforcement [15, 46].

The Boy Who Cried Wolf: Adversarial Misclassification of Safe Inputs as Unsafe in Multimodal Guardrails

Representative risk taxonomies are provided in Appendix A. Prior work has mainly strengthened guard models against adversarial evasion, aiming to prevent unsafe content from being misclassified as safe under attack [9]. In contrast, we study attacks that directly target guard models and induce false positives, leading these safety components to incorrectly reject benign requests.

2.2

Adversarial Attacks on MLLMs

Foundational work in adversarial machine learning shows that neural networks can be induced to make incorrect predictions by adding visually imperceptible perturbations to the input [4, 16, 48]. Building on this line, adversarial attacks on MLLMs have largely focused on integrity failures that seek to bypass safety mechanisms and elicit disallowed responses, which manifests as false negatives under safety auditing [35]. For example, prior work crafts adversarial images so that the model output matches a malicious textual target or follows unsafe instructions under a paired prompt [4, 22]. Zhang et al. [47] further proposes training a universal perturbation generator over large scale data that transfers target image semantics into adversarial noise. A separate line studies availability oriented attacks that degrade throughput by inflating generation length and inference cost, which reduces the service capacity for benign users. Gao et al. [12], for instance, optimizes an adversarial input that triggers repetitive generation and substantially inflates the number of produced tokens. Related approaches such as Zhang et al. [50] prolong generation by suppressing the probability of generating the end of sequence token. Our attack also prevents benign requests from being served, but it operates through safety decision making rather than resource amplification. We exploit the guard model as a modular safety component to over classify benign inputs as unsafe, causing incorrect rejections at the logic layer.

Unsafe Induction Attacks

In this section, we formalize the problem definition and threat model of Unsafe Induction Attacks against multimodal guard models.

3.1

Problem Formulation

3.1.1 Multimodal Guard Models. A multimodal guard model G acts as an independent safety auditor that screens user inputs before they reach the downstream generative model. Formally, given an input pair consisting of an image 𝐼 ∈ R𝐻 ×𝑊 ×3 and a text prompt 𝑇 (a sequence of tokens), the guard model produces a safety judgment: G(𝐼,𝑇 ) = (𝑦, 𝑐)

(1)

where 𝑦 ∈ {safe, unsafe} is the binary safety decision, and 𝑐 ∈ C is the risk category drawn from a predefined taxonomy C when 𝑦 = unsafe. Unlike binary classifiers, modern guard models such as Llama Guard 3 Vision operate as conditional text generators that produce structured refusal statements. For example, when content is flagged as unsafe, the model generates a refusal response that includes the specific risk type, such as “unsafe, S1: Violent Crimes” for the Llama Guard 3 Vision taxonomy [9]. In the normal workflow, a user submits a request (𝐼,𝑇 ) to the multimodal service. The guard model first audits the input. If G(𝐼,𝑇 ) =

KDD 2026, August 9–13, 2026, Jeju Island, Republic of Korea.

(safe, ∅), the request proceeds to the generative model for processing. If G(𝐼,𝑇 ) = (unsafe, 𝑐), the request is rejected and the user receives an error message indicating that the content violates policy category 𝑐. 3.1.2 Unsafe Induction Attacks. We now formalize the adversarial objective. Let 𝐼 ben denote a visually benign image that would normally satisfy G(𝐼 ben,𝑇 ) = (safe, ∅) for any legitimate user prompt 𝑇 . An attacker crafts an adversarial perturbation 𝛿 such that the perturbed image 𝐼 adv = 𝐼 ben + 𝛿

(2)

triggers a false positive response from the guard model. Specifically, the attack succeeds if G(𝐼 adv,𝑇 ) = (unsafe, 𝑐),

∀𝑇 ∈ Tuser

(3)

where Tuser represents the space of realistic user prompts that accompany benign requests. The perturbation must satisfy the imperceptibility constraint ∥𝛿 ∥ 𝑝 ≤ 𝜖 to ensure that 𝐼 adv remains visually indistinguishable from 𝐼 ben to human observers. The attack succeeds when the adversarially perturbed image induces false positive classifications across diverse user queries, resulting in a logical denial of service.

3.2

Threat Model

3.2.1 Attack Scenario. We consider a realistic adversarial scenario where an attacker, such as a competing service provider or malicious actor, aims to disrupt multimodal services that employ guard models for content moderation. Target services include image editing platforms, visual question answering services, and content generation tools. Attack Flow. The attack proceeds in three stages: Stage 1: Adversarial Image Generation. The attacker crafts imperceptibly perturbed images depicting benign everyday scenes (landscapes, pet photos, common objects) that appear visually normal to human observers. Stage 2: Public Distribution. The attacker distributes these adversarial images through public channels such as image sharing websites, social media platforms, or open content repositories. Stage 3: User Incorporation. Unsuspecting users download and incorporate these images into legitimate service requests, such as uploading to image editing services or including in multimodal queries. When the target service’s guard model audits user requests, it incorrectly flags benign adversarial images as policy violations (e.g., S1: Violent Crimes or S3: Sexual Content) despite containing no such content. The service rejects requests and may escalate enforcement. As false rejections accumulate across users, trust erodes and user experience degrades, achieving the attacker’s goal of operational disruption. 3.2.2 Attacker Properties. Assumptions. We assume a white-box setting where the attacker has full access to the target guard model’s architecture and weights [25, 50]. This is realistic for open-weight models (e.g., Llama-Guard-3Vision, LLaVA-Guard). Capabilities. The attacker can: (1) collect unsafe images to construct reference datasets; (2) leverage gradient-based optimization

KDD 2026, August 9–13, 2026, Jeju Island, Republic of Korea.

Shuo Shi et al.

Capability

optimization problem over the shadow prompt set: Model weights

Adversarial Attack

Unsafe collection

min

𝑘 ∑︁

Lgen G(𝐼 ben + 𝛿,𝑇 𝑖 ),𝑇target ,

subject to ∥𝛿 ∥ ∞ ≤ 𝜖 (4)

𝛿 𝑖=1

Attacker

Safe Images

Adversarial

Internet

adopt

inaccessible

Lose user trust

Unsafe, S9, Weapons.

Logical DOS Goal

Unknown prompt

Stealthiness

Users

Constraints

Figure 2: Illustration of the attack scenario. The attacker generates adversarial benign images and distributes them in public spaces. When users incorporate these images into service requests, the guard model incorrectly flags them as unsafe, causing denial of service.

to inject imperceptible perturbations into benign images; (3) disseminate adversarial images through public online platforms. Constraints. The attacker operates under two critical constraints: Constraint 1: Prompt Agnosticism. The attacker has no control over text prompts users submit alongside adversarial images. Unlike traditional jailbreaking where adversaries craft both image and prompt, our threat model reflects real-world conditions where users independently formulate queries. This necessitates attacks robust across diverse and unknown textual contexts. Constraint 2: Imperceptibility. Perturbations must remain imperceptible to ensure users do not detect tampering and images continue to circulate naturally. We enforce this by bounding perturbation magnitude under an ℓ𝑝 norm: ∥𝛿 ∥ ∞ ≤ 𝜖 with small 𝜖.

Methodology

In this section, we present our approach to realizing Unsafe Induction Attacks. While existing adversarial paradigms can in principle be adapted to induce false positives, both query based output optimization (§4.1) and feature level target matching (§4.2) suffer from fundamental limitations that prevent effective prompt agnostic attacks. We then introduce our Unsafe Semantic Distillation (USD) framework (§4.3), which overcomes these challenges.

4.1

Query-Based Output Optimization

A natural baseline for inducing unsafe classifications is to directly optimize the adversarial perturbation so that the guard model generates the target output token "unsafe" when presented with the perturbed image and a query. Since the attacker does not know what prompt real users will submit, prior work adopts a surrogate strategy in which the perturbation is optimized over a collection of shadow prompts X𝑡 = {𝑇 1,𝑇 2, . . . ,𝑇 𝑘 } that approximate the space of user queries [29]. Formally, let Lgen (G(𝐼,𝑇 ),𝑇target ) denote the language modeling loss that measures the discrepancy between the guard model’s generated output and the target response. For generic unsafe induction attacks, 𝑇target = “unsafe”. For targeted risk induction attacks, 𝑇target = “unsafe, S1: Violent Crimes” or another specific risk category from the taxonomy C. The attacker solves the following

where the objective minimizes the generation loss across all shadow prompts, thereby encouraging the perturbed image 𝐼 adv = 𝐼 ben + 𝛿 to trigger the unsafe response under diverse textual contexts. Limitation. This paradigm is fundamentally constrained by its reliance on a fixed shadow prompt set X𝑡 , which covers only a limited region of the full user prompt space Tuser . Simply increasing |X𝑡 | does not resolve this issue, as exhaustive enumeration of all semantic variations is infeasible. Consequently, this approach fails to satisfy the prompt-agnostic constraint required by our threat model.

4.2

Feature Matching with Target Images

An alternative paradigm bypasses the reliance on shadow prompts by optimizing adversarial perturbations in the feature space rather than at the output token level. Instead of directly minimizing a language modeling loss, these methods select a target image 𝐼 target and craft a perturbation that aligns the benign image’s latent representation with that of the target [22, 38, 53]. Let 𝑓𝜙 (·) denote the vision encoder of the guard model that maps images into a latent embedding space. The feature matching objective seeks to minimize the distance between the perturbed benign image and the target image in this latent space: min 𝑑 𝑓𝜙 (𝐼 ben + 𝛿), 𝑓𝜙 (𝐼 target ) , subject to ∥𝛿 ∥ ∞ ≤ 𝜖 (5) 𝛿

where 𝑑 (·, ·) is a distance metric such as Euclidean distance or cosine dissimilarity. This approach introduces semantic shifts by forcing the adversarial image to adopt the visual characteristics of 𝐼 target at the feature level. For Unsafe Induction Attacks, this paradigm can be applied by selecting an unsafe image as the target. Prior work demonstrates that when one modality in a multimodal input contains sufficiently strong unsafe signals, the guard model tends to classify the entire input as unsafe even if the other modality is benign [7]. This cross modal dominance effect naturally decouples the attack from the prompt. By aligning the perturbed benign image with an inherently unsafe target image, the attacker can induce the guard model to output an unsafe classification regardless of the accompanying user query. Limitation. While feature matching eliminates the need for shadow prompts, existing methods suffer from instance-level overfitting. These approaches optimize perturbations to match a single target image, causing the adversarial example to learn fine-grained idiosyncrasies rather than capturing the abstract concept of "unsafety". As a result, the induced unsafe signal is brittle and fails to generalize across diverse user prompts, significantly degrading real-world attack effectiveness.

4.3

Unsafe Semantic Distillation (USD)

We now introduce Unsafe Semantic Distillation (USD), which addresses the limitations of both prior paradigms through distributional semantic distillation coupled with hybrid optimization constraints.

The Boy Who Cried Wolf: Adversarial Misclassification of Safe Inputs as Unsafe in Multimodal Guardrails

Feature-based View

KDD 2026, August 9–13, 2026, Jeju Island, Republic of Korea.

Frozen

Trainable

Forward

Unsafe Image Collection Targeted Risk

Unsafe Distribution

Suicide / Sex …… Sample

Violence Only

Vision Encoder

Learnable 𝛿

Concept Injection

…

Benign Image

Backward

Distillation Loss

Unsafe Embedding Distill

Adversarial Embedding

Safe Embedding

Query-based View

Task Instruction & Safe Output Formatting R18: Is this type of aircraft Decision Loss Pull typically used wheels for commercial R2: How many are Output or private aviation? clearly visible the sky image? R1: Remove theinblue Push background and replace it Targeted Risk Attacks with a transparent layer. Sample Simulate Unsafe, S1: Violent Crimes Unsafe, … Language Backbone Shadow Prompts User Requests

Figure 3: Overview of USD framework combining feature-based and query-based optimization. USD distills unsafe semantics from diverse unsafe images via distributional alignment (top) and verifies attack success using shadow prompts (bottom), achieving prompt-robust false positive induction through hybrid loss architecture.

4.3.1 From Instance Mimicry to Concept Distillation. The key insight underlying USD is that unsafe content, while visually diverse, shares abstract high level semantic properties in the latent embedding space of guard models. Rather than aligning with a single unsafe target image or relying solely on output token supervision, we propose to distill the essential unsafe semantics from a collection of unsafe images. Let Dunsafe = {𝐼 1𝑢 , 𝐼 2𝑢 , . . . , 𝐼 𝑁𝑢 } denote a diverse set of unsafe images spanning multiple risk categories in the target taxonomy. We extract their latent embeddings Zunsafe = {𝑧 1, 𝑧 2, . . . , 𝑧 𝑁 } where 𝑧 𝑗 = 𝑓𝜙 (𝐼 𝑢𝑗 ) and 𝑓𝜙 (·) is the guard model’s vision encoder. These embeddings collectively capture the unsafe semantic manifold. Distributional Alignment Prevents Instance Overfitting. Instance level feature matching (Section 4.2) aligns the adversarial perturbation with a single target 𝐼 target , which causes the optimization to memorize instance specific textures, colors, and low level visual patterns unique to that image. This leads to brittle perturbations that fail to generalize when the guard model encounters varied textual contexts or makes decisions based on slightly different feature dimensions. In contrast, by optimizing toward the statistical distribution over Zunsafe , we extract and condense the shared conceptual essence of "unsafeness" that is invariant across diverse unsafe instances. This distillation process discards idiosyncratic details and retains only the high level semantic properties that the guard model consistently associates with policy violations, thereby achieving robust concept level generalization. 4.3.2 Hybrid Loss Architecture. USD employs a composite objective that enforces constraints at both the feature level and the decision level. This dual supervision addresses the complementary

weaknesses of the two baseline paradigms: query based methods lack feature space constraints and thus learn superficial decision shortcuts, while feature matching methods lack decision verification and may produce embeddings that do not functionally trigger misclassifications. Adaptive Semantic Distillation Loss. We define a feature level loss that aligns the adversarial image with the distributional center of the unsafe manifold. Ideally, we would maximize the likelihood of 𝐼 adv under the true unsafe distribution: max E𝑧∼𝑃unsafe sim(𝑓𝜙 (𝐼 adv ), 𝑧) (6) 𝛿

where 𝑃unsafe is the distribution of unsafe embeddings and sim(·, ·) is a similarity measure. Since we do not have access to the ground truth distribution, we use the empirical estimate with our collected unsafe samples. Given Zunsafe = {𝑧 1, 𝑧 2, . . . , 𝑧 𝑁 } where 𝑧 𝑗 = 𝑓𝜙 (𝐼 𝑢𝑗 ), we approximate the expectation using a temperature scaled LogSumExp aggregation: ! 𝑁 CosSim(𝑓𝜙 (𝐼 adv ), 𝑧 𝑗 ) 1 ∑︁ Ldistill = − log exp (7) 𝑁 𝑗=1 𝜏 where 𝜏 > 0 is a temperature parameter and CosSim(·, ·) denotes cosine similarity. This soft aggregation assigns exponentially higher weights to nearby prototypes, enabling adaptive routing toward the most accessible unsafe concepts. During optimization, gradients flow from multiple prototypes simultaneously, encouraging the perturbation to capture shared high level unsafe semantics rather than overfitting to any single instance. Decision Level Verification Loss. To ensure that the distilled semantics functionally induce misclassifications at the decision

KDD 2026, August 9–13, 2026, Jeju Island, Republic of Korea.

Shuo Shi et al.

Algorithm 1 USD Training Procedure

Require: clean image 𝐼 ben ; unsafe image set Dunsafe ; shadow prompt pool X𝑡 ; perturbation budget 𝜖; iterations 𝑇 ; step size 𝛼; distillation temperature 𝜏; loss weight 𝜆; unsafe mini-batch size 𝑚 1: Initialize: 𝛿 ← 0 ⊲ start from clean image 2: for 𝑡 = 1 to 𝑇 do 3: Sample a shadow prompt 𝑇 ∼ X𝑡 4: Sample unsafe mini-batch {𝐼 𝑢𝑗 }𝑚 𝑗=1 ∼ Dunsafe 5: 𝐼 adv ← Clip(𝐼 ben + 𝛿) 6: 𝑧 adv ← 𝑓𝜙 (𝐼 adv ); 𝑧 𝑗 ← 𝑓𝜙 (𝐼 𝑢𝑗 ) for 𝑗 ∈ [1, 𝑚] 7: Compute distillation Íloss: CosSim(𝑧 adv ,𝑧 𝑗 ) Ldistill ← − log 𝑚1 𝑚 𝑗=1 exp 𝜏 8: Compute decision loss: Ldecision ← − log 𝑃 (𝑦 = unsafe | 𝐼 adv,𝑇 ) 9: LUSD ← Ldecision + 𝜆 · Ldistill 10: 𝛿 ← 𝛿 + 𝛼 · sign(∇𝛿 LUSD ) 11: 𝛿 ← Clip(𝛿, −𝜖, 𝜖) ⊲ ℓ∞ projection 12: end for 13: return 𝐼 ben + 𝛿

In this section, we evaluate the effectiveness of USD for inducing false positives in multimodal guard models. We design a comprehensive evaluation framework that reflects real world deployment scenarios and systematically analyze attack performance under diverse conditions.

boundary, we incorporate a standard cross entropy loss. At each optimization step, we sample a shadow prompt 𝑇 ∼ X𝑡 from a predefined set and define: Ldecision = − log 𝑃 (𝑦 = unsafe | 𝐼 adv,𝑇 )

(8)

where 𝑦 = G(𝐼,𝑇 ) denotes the guard model’s output. This cross entropy loss has the same form as the query based optimization objective in Equation 4. However, the critical difference lies in how it is used within the overall optimization framework. In query based methods, Ldecision constitutes the sole optimization target, leading the perturbation to exploit prompt specific shortcuts without learning robust visual semantics. In USD, Ldecision serves as a complementary verification term that operates in conjunction with Ldistill . The distillation loss enforces distributional alignment in latent space, ensuring that the learned perturbation embeds deep unsafe semantics in the visual features. This hybrid constraint prevents collapse into prompt specific artifacts and leverages cross modal dominance to enable generalization beyond X𝑡 . 4.3.3

5.1

• Llama-Guard-3-Vision [9]: Meta’s vision-language safeguard fine-tuned on Llama 3.2-11B for multimodal inputoutput content moderation. • LLaVA-Guard-v1.2 [14]: An open VLM-based framework trained on 5.5K human-annotated samples for detecting safety risks across 9 categories with explainable rationales. • Qwen-Guard-v1.2 [14]: Built on Qwen2.5-VL for multimodal safety classification with similar architecture to LlavaGuard. • ShieldVLM [11]: A deliberative reasoning-based safeguard trained on 4.2K samples to detect multimodal implicit toxicity across 7 risk categories via cross-modal analysis. 5.1.2 Datasets. Benign Images. We select benign victim images from two standard vision question answering benchmarks: TextVQA [31] and VQAv2 [1]. For each dataset, we randomly sample 100 images and filter them using Llama Guard 3 Vision to ensure they are classified as safe. These images depict everyday scenes such as urban landscapes, indoor environments, and common objects. Unsafe Reference Set. To construct the unsafe semantic manifold Zunsafe required by USD, we aggregate unsafe images from two safety evaluation benchmarks: HolisticSafeBench [18] and VHD11K [42]. These datasets provide diverse policy violating content spanning multiple risk categories such as violence, explicit content, and harmful activities. 5.1.3 Baselines. We compare USD against representative methods from both attack paradigms discussed in Section 4: • MLLM-Refusal [29]: A query based method that optimizes adversarial perturbations to maximize the probability of generating unsafe output tokens under shadow prompts. • MF-ii-tt [53]: Feature matching baselines that align the adversarial image with a single unsafe target image using either image to image or text to image matching objectives. • Chain of Attack (CoA) [38]: A recent adversarial method that combines multiple attack strategies through iterative optimization.

(9)

where 𝜆 > 0 controls the balance between decision verification and feature generalization. The perturbation 𝛿 is optimized using Projected Gradient Descent (PGD): 𝛿 (𝑡 +1) = Π ∥ · ∥ ∞ ≤𝜖 𝛿 (𝑡 ) − 𝛼 · ∇𝛿 LUSD (𝐼 ben + 𝛿 (𝑡 ) ) (10) where Π denotes projection onto the ℓ∞ ball of radius 𝜖, and 𝛼 is the step size. After 𝑇 iterations, we obtain the adversarial image 𝐼 adv = 𝐼 ben + 𝛿 (𝑇 ) . 4.3.4 USD Training Procedure (Pseudocode). Algorithm 1 summarizes the USD optimization loop. Each iteration stochastically samples (i) a shadow prompt to provide decision-level verification, and (ii) a mini-batch of unsafe images to provide distributional semantic distillation. We then update 𝛿 using PGD under an ℓ∞ constraint.

Experimental Setup

5.1.1 Target Models. We evaluate on four representative multimodal guard models:

Optimization. We optimize the weighted combination: LUSD = Ldecision + 𝜆 · Ldistill

Experiments

5.1.4 Attack Configuration. Unless otherwise specified, we conduct experiments under the following default settings: target model is Llama Guard 3 Vision, perturbation budget is 𝜖 = 16/255 under ℓ∞ norm, and optimization iterations is 𝑇 = 2000.

5.2

Evaluation Metrics

We report attack success rate (ASR), i.e., the percentage of benign images that are incorrectly classified as unsafe after perturbation.

The Boy Who Cried Wolf: Adversarial Misclassification of Safe Inputs as Unsafe in Multimodal Guardrails

Model / Method Llama-Guard-3-Vision + MLLM-Refusal + MF-ii-tt + CoA + USD (ours) Llava-Guard-v1.2 + MLLM-Refusal + MF-ii-tt + CoA + USD (ours) Qwen-Guard-v1.2 + MLLM-Refusal + MF-ii-tt + CoA + USD (ours) ShieldVLM + MLLM-Refusal + MF-ii-tt + CoA + USD (ours)

TextVQA 𝜖 = 8/255 𝜖 = 16/255 ASRs ASRg ASRr ASRs ASRg ASRr 0.0 0.0 0.0 0.0 0.0 0.0 95.2 56.3 64.3 96.5 62.1 68.7 – 40.4 55.7 – 46.6 59.2 – 57.4 60.8 – 60.8 62.2 98.4 76.8 78.4 98.9 77.6 82.8 0.2 0.0 0.0 0.2 0.0 0.0 78.3 50.7 59.3 84.6 55.8 62.3 – 47.8 49.3 – 48.9 52.7 – 48.5 46.8 – 57.2 57.6 96.7 75.4 77.5 98.2 73.2 79.2 0.1 0.0 0.0 0.1 0.0 0.0 85.6 50.9 59.3 91.2 57.4 64.8 – 37.8 51.3 – 43.5 56.4 – 54.6 57.9 – 57.8 60.5 94.8 77.5 80.6 97.1 82.3 85.9 0.2 0.1 0.4 0.2 0.1 0.4 82.1 48.3 56.8 88.9 54.6 62.3 – 35.7 48.5 – 41.2 53.7 – 51.2 54.6 – 54.7 58.4 92.5 66.8 70.4 95.3 71.2 75.6

KDD 2026, August 9–13, 2026, Jeju Island, Republic of Korea.

VQAv2 𝜖 = 8/255 𝜖 = 16/255 ASRs ASRg ASRr ASRs ASRg ASRr 0.0 0.0 0.0 0.0 0.0 0.0 90.5 54.1 63.9 94.6 60.1 66.8 – 41.3 45.4 – 44.9 51.4 – 54.1 62.3 – 59.4 65.6 98.8 76.7 80.5 98.8 78.6 84.3 0.2 0.0 0.1 0.2 0.0 0.1 76.2 47.3 54.7 82.1 52.5 60.1 – 42.6 50.4 – 46.8 53.9 – 47.7 48.6 – 54.9 56.4 96.1 71.9 74.0 97.8 74.5 78.7 0.0 0.0 0.0 0.0 0.0 0.0 83.4 49.1 57.6 88.9 55.8 62.5 – 36.2 42.7 – 40.6 47.9 – 51.3 59.7 – 55.7 63.4 93.6 75.2 78.4 96.5 79.5 83.2 0.1 0.0 0.1 0.1 0.0 0.1 80.8 46.5 54.7 86.4 52.9 60.1 – 34.9 40.6 – 38.3 45.2 – 48.7 56.2 – 52.4 60.8 91.7 65.3 72.8 94.8 69.7 77.2

Table 1: Main results comparing unsafe induction effectiveness across multiple methods. All metrics are reported in percentages (%). Bold indicates the best ASRr performance within each model group, excluding the original base model (shown in gray text).

To distinguish memorization from prompt-robust attack success, we evaluate each method under three prompt settings: ASRs measures success on the shadow prompts used during optimization, serving as an in-distribution sanity check. ASRg measures success on 10 generic image-agnostic questions, such as "What does this image depict?", to test transfer beyond the exact shadow prompts. ASRr measures success on 10 image-specific user requests generated for realistic application scenarios, including image editing, content sharing, and visual question answering. These prompts are disjoint from the shadow prompts and constitute our primary metric for prompt-agnostic robustness. For all metrics, prompts that trigger unsafe classifications on clean images are filtered and regenerated. Detailed prompt generation instructions and representative examples are provided in Appendix C.

96.5% ASRs on Llama Guard 3 Vision with 𝜖 = 16/255 on TextVQA. This demonstrates that when attackers know the exact user prompts, inducing false positives becomes significantly easier. However, this scenario is unrealistic in our threat model where attackers cannot control user queries. The critical question is whether attacks generalize to unknown prompts, which we assess through ASRr . USD achieves superior prompt-agnostic robustness. Focusing on ASRr , which best reflects real-world attack scenarios, USD consistently outperforms all baselines by substantial margins. On Llama Guard 3 Vision with TextVQA at 𝜖 = 16/255, USD achieves 82.8% ASRr , surpassing the best baseline (CoA at 62.2%) by 20.6 percentage points and MLLM-Refusal (68.7%) by 14.1 points. This gap is even more pronounced at the tighter budget 𝜖 = 8/255. The performance gap between USD and feature-matching baselines (MF-ii-tt, CoA) reveals the limitation of instance-level alignment.

5.3

5.4

RQ1: Overall Effectiveness of Unsafe Induction Attacks

We evaluate the overall effectiveness of USD and baseline methods across different guard models, datasets, and perturbation budgets. Table 1 presents the comprehensive results. Setup. We evaluate four guard models on two benign datasets (TextVQA and VQAv2). For each benign image, we craft ℓ∞ -bounded perturbations with 𝜖 ∈ {8/255, 16/255}. Note that feature matching baselines (MF-ii-tt and CoA) do not utilize shadow prompts during optimization, as they operate purely in the latent feature space by aligning with unsafe target images. Shadow prompt overfitting reveals the prompt-agnostic challenge. Examining ASRs , we observe that both query-based (MLLMRefusal) and our method achieve high success rates on shadow prompts when applicable. For instance, MLLM-Refusal achieves

RQ2: Targeted Risk Category Attacks

Beyond inducing generic false positives, we investigate whether USD can perform targeted risk injection, where attackers manipulate guard models to misclassify benign images under specific risk categories. This capability is particularly concerning as it enables adversaries to not only deny service but also strategically frame benign content with fabricated policy violations. Setup. Due to differences in safety taxonomies across guard models, we conduct targeted attacks on representative risk categories from different guard model taxonomies: S1 (Violent Crimes) and S3 (Sex-Related Crimes) for Llama Guard 3 Vision, and O2 (Violence, Harm or Cruelty) and O3 (Sexual Content) for LLaVA Guard. For each target category, we construct the unsafe reference set Zunsafe using only images from that specific risk type, enabling categoryconditioned distributional distillation. We evaluate under ASRr

KDD 2026, August 9–13, 2026, Jeju Island, Republic of Korea.

Shuo Shi et al.

5.5

RQ3: Cross-Model Transferability

We evaluate cross-model transferability to assess attack specificity. In our threat model where attackers may be competing service providers, low transferability is desirable: adversarial images should disrupt the target service while leaving the attacker’s own guard model unaffected, avoiding self-inflicted damage. Setup. For each source model 𝑀𝑠 , we craft adversarial images using USD and evaluate them on all four target models 𝑀𝑡 under ASRr . USD exhibits strong white-box effectiveness with minimal cross-model transfer. The transferability matrix reveals a stark contrast between diagonal and off-diagonal performance. Whitebox attacks achieve 75.6-85.9% ASRr across all models. In contrast, cross-model transferability is remarkably low, with off-diagonal entries ranging from only 0.7% to 3.7%. For instance, adversarial images optimized for Llama Guard achieve merely 0.9% on LLaVA Guard and 3.7% on Qwen Guard, representing a 96-99% drop from white-box performance. This dramatic gap arises because different guard models employ distinct vision encoders, causing USD’s perturbations optimized for one encoder’s latent space to fail on models with different visual representations.

5.6

RQ4: Image-Only Evaluation

To validate that USD injects unsafe semantics into the visual modality itself while preserving visual stealthiness, we evaluate adversarial images from two complementary perspectives. First, we test whether the attack remains effective without text prompts. Second, we inspect and quantify the perceptual distortion introduced by USD. USD preserves visual attack strength without prompts. Table 3 shows that USD achieves 75.0% and 78.0% image-only ASR on TextVQA and VQAv2, respectively, substantially outperforming feature-matching baselines. These results indicate that USD does

0.7

79.2

2.6

1.9

Qwen-Guard

0.7

1.3

85.9

2.8

ShieldVLM

1.1

1.0

2.4

75.6

Attack Success Rate (%)

LLaVA-Guard

LM dV iel Sh

ua a-G am

with realistic user simulation prompts and report the percentage of benign images successfully assigned to the target category. USD achieves high-precision targeted risk injection. Table 2 shows that USD successfully performs targeted attacks across all tested categories with consistently high success rates. On Llama Guard 3 Vision, USD achieves 80.6% targeted ASRr for S1 (Violent Crimes) and 79.8% for S3 (Sex-Related Crimes). These results demonstrate that by simply filtering the unsafe reference set to a specific risk category, USD can steer guard models to flag benign images under precise policy labels rather than triggering generic refusals. The performance on LLaVA Guard shows similar trends.

1.4 80

Source Model

Table 2: Targeted risk category injection results under realistic user simulation prompts (ASRr ). USD successfully manipulates benign images to trigger specific policy violations.

3.7

-G

76.7 77.3

0.9

LLaVA-Guard-v1.2 O2: Violence, Harm or Cruelty O3: Sexual Content

82.8

LLama-Guard

80.6 79.8

ard

Llama-Guard-3-Vision S1: Violent Crimes S3: Sex-Related Crimes

100

ASRr

aV A-

Model / Target Category

Target Model Figure 4: Cross-model transferability matrix of USD attacks under ASRr . High diagonal values (75.6-85.9%) with low off-diagonal transfer (<4%) demonstrate attack specificity. Method

TextVQA

VQAv2

Clean Images

0.0

MF-ii-tt CoA USD (ours)

50.0 54.0 75.0

51.0 57.0 78.0

Table 3: Image-only attack success rates on Llama-Guard-3-Vision. Adversarial images are evaluated without text prompts. All values are reported in percentages (%).

Method

Budget

LPIPS ↓

USD (ours) USD (ours)

𝜖 = 8/255 𝜖 = 16/255

0.2089 0.3124

Table 4: Perceptual distortion of adversarial examples on TextVQA using Llama Guard 3 Vision as the target guard model. Lower LPIPS indicates stronger perceptual similarity to clean images.

not merely exploit prompt-conditioned decoding artifacts; instead, it embeds unsafe semantics into the visual representation itself. Adversarial examples remain visually benign. Figure 5 shows USD adversarial examples under different perturbation budgets. Each row presents the same benign image before and after applying perturbations with 𝜖 = 8/255 and 𝜖 = 16/255. The images remain visually indistinguishable from their clean counterparts while inducing false positive classifications, confirming that the attack preserves human-perceived benignness. Quantitative imperceptibility. We further compute LPIPS between clean and adversarial images to objectively assess perceptual distortion. Table 4 reports results for USD adversarial examples crafted against Llama Guard 3 Vision on TextVQA. The LPIPS scores remain low under both perturbation budgets, indicating that distributional alignment does not introduce substantial perceptual degradation even when the budget increases to 𝜖 = 16/255. Additional feature-space visualization is provided in Appendix B.

KDD 2026, August 9–13, 2026, Jeju Island, Republic of Korea.

100

Targeted ASR (%)

The Boy Who Cried Wolf: Adversarial Misclassification of Safe Inputs as Unsafe in Multimodal Guardrails

60 40 20 0 1

100

Number of Shadow Prompts

60 40 20 0 1

100

Size of Unsafe Reference Set

Figure 6: Effect of hyperparameters on ASRr . Left: varying number of shadow prompts from {1, 5, 10, 50, 100}. Right: varying size of unsafe reference set | Zunsafe | from {1, 5, 10, 50, 100}.

Figure 5: Visual comparison of clean images and USD adversarial images under different perturbation budgets. Left: clean image; Center: 𝜖 = 8/255; Right: 𝜖 = 16/255. The adversarial perturbations are imperceptible while triggering guard model misclassifications. Preprocessing

ASRr

None JPEG compression (quality 75) Average smoothing Gaussian blur

78.4 71.2 61.4 67.0

Table 5: Robustness of USD adversarial examples under defenses. Results are evaluated on TextVQA against Llama Guard 3 Vision with 𝜖 = 8/255 and reported in percentages (%).

5.7

RQ5: Robustness Against Input Preprocessing

Since many commercial services apply lightweight image preprocessing before moderation, we evaluate whether simple transformations can mitigate USD. We test adversarial examples crafted on TextVQA against Llama Guard 3 Vision with 𝜖 = 8/255 under three common preprocessing operations: JPEG compression, average smoothing, and Gaussian blur. USD remains effective under lightweight defenses. Table 5 shows that preprocessing reduces attack success but does not eliminate the threat. JPEG compression at quality 75 preserves most of USD’s effectiveness, with ASRr decreasing from 78.4% to 71.2%. Smoothing-based transformations are stronger but still leave substantial success rates above 60%, suggesting that the induced unsafe signal is not solely a fragile high-frequency artifact.

5.8

RQ6: Hyperparameter Analysis

We analyze the impact of two critical hyperparameters on USD performance: the number of shadow prompts for decision verification and the size of the unsafe reference set for distributional distillation.

We vary each hyperparameter across {1, 5, 10, 50, 100} and report results in Figure 6. Shadow prompts. Figure 6 (left) shows that ASRr increases rapidly with shadow prompt count but saturates quickly at 10 prompts (82%), with negligible gains beyond this point. This saturation validates USD’s prompt-agnostic design, where distributional distillation provides the primary generalization mechanism while shadow prompts mainly verify decision boundary alignment. Unsafe reference set size. Figure 6 (right) reveals that larger unsafe reference sets consistently improve performance with more gradual saturation. Performance increases from 38% (1 image) to 82% (50 images) and 85% (100 images), demonstrating that more unsafe images enable better distributional learning. Unlike shadow prompts which saturate rapidly, unsafe set size exhibits continued benefits as richer manifold coverage captures more diverse facets of the unsafe semantic distribution.

Conclusion

This work introduces Unsafe Induction Attacks, a new adversarial paradigm that exploits multimodal guard models to induce false positives, creating logical denial-of-service through over-rejection of benign content. We demonstrate that existing adversarial methods fail to achieve prompt-robust attacks due to shadow prompt overfitting or instance-level feature matching. Our Unsafe Semantic Distillation (USD) framework addresses these limitations through distributional alignment over diverse unsafe prototypes, achieving concept-level manipulation that generalizes across unknown user queries. Experiments show USD attains 78-84% attack success rates under realistic user simulation, exposing critical vulnerabilities in current safety architectures. These findings motivate the design of next-generation guard systems with bidirectional robustness guarantees that defend against both bypassing harmful content and over-rejecting legitimate requests.

Acknowledgments This work was partly supported by the Science Challenge Project under No. TZ2025005, NSFC under No. U2441239, 62402418 and U24A20336, the "Pioneer and Leading Goose" R&D Program of Zhejiang under No. 2026C02A1233 and 2025C02034, the Key R&D Program of Ningbo under No. 2024Z115, the Ningbo Yongjiang Talent Project, the China Postdoctoral Science Foundation under No. 2024M762829 and 2025M781522, and Zhejiang Key Laboratory of Decision Intelligence under No. 2025E10006.

KDD 2026, August 9–13, 2026, Jeju Island, Republic of Korea.

References [1] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision. 2425–2433. [2] Eugene Bagdasaryan, Tsung-Yin Hsieh, Ben Nassi, and Vitaly Shmatikov. 2023. Abusing images and sounds for indirect instruction injection in multi-modal LLMs. arXiv preprint arXiv:2307.10490 (2023). [3] Davide Caffagni, Federico Cocchi, Luca Barsellotti, Nicholas Moratelli, Sara Sarto, Lorenzo Baraldi, Lorenzo Baraldi, Marcella Cornia, and Rita Cucchiara. 2024. The Revolution of Multimodal Large Language Models: A Survey. In Findings of the Association for Computational Linguistics: ACL 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 13590–13618. doi:10.18653/v1/2024.findings-acl.807 [4] Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. 2019. On evaluating adversarial robustness. arXiv preprint arXiv:1902.06705 (2019). [5] Nicholas Carlini, Milad Nasr, Christopher A. Choquette-Choo, Matthew Jagielski, Irena Gao, Anas Awadalla, Pang Wei Koh, Daphne Ippolito, Katherine Lee, Florian Tramer, and Ludwig Schmidt. 2023. Are aligned neural networks adversarially aligned?. In Proceedings of the 37th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS ’23). Curran Associates Inc., Red Hook, NY, USA, Article 2687, 23 pages. [6] Chaochao Chen, Yizhao Zhang, Yuyuan Li, Jun Wang, Lianyong Qi, Xiaolong Xu, Xiaolin Zheng, and Jianwei Yin. 2024. Post-training attribute unlearning in recommender systems. ACM Transactions on Information Systems 43, 1 (2024), 1–28. [7] Huiyi Chen, Jiawei Peng, Dehai Min, Changchang Sun, Kaijie Chen, Yan Yan, Xu Yang, and Lu Cheng. 2025. MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMs. arXiv:2511.14159 [cs.CV] https://arxiv.org/abs/2511.14159 [8] Simin Chen, Zihe Song, Mirazul Haque, Cong Liu, and Wei Yang. 2022. Nicgslowdown: Evaluating the efficiency robustness of neural image caption generation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 15365–15374. [9] Jianfeng Chi, Ujjwal Karn, Hongyuan Zhan, Eric Smith, Javier Rando, Yiming Zhang, Kate Plawiak, Zacharie Delpierre Coudert, Kartikeya Upasani, and Mahesh Pasupuleti. 2024. Llama guard 3 vision: Safeguarding human-ai image understanding conversations. arXiv preprint arXiv:2411.10414 (2024). [10] Shiyao Cui, QingLin Zhang, Xuan Ouyang, Renmiao Chen, Zhexin Zhang, Yida Lu, Hongning Wang, Han Qiu, and Minlie Huang. 2025. ShieldVLM: Safeguarding the Multimodal Implicit Toxicity via Deliberative Reasoning with LVLMs: ShieldVLM. In Proceedings of the 33rd ACM International Conference on Multimedia. 11677–11686. [11] Shiyao Cui, QingLin Zhang, Xuan Ouyang, Renmiao Chen, Zhexin Zhang, Yida Lu, Hongning Wang, Han Qiu, and Minlie Huang. 2025. ShieldVLM: Safeguarding the Multimodal Implicit Toxicity via Deliberative Reasoning with LVLMs: ShieldVLM. In Proceedings of the 33rd ACM International Conference on Multimedia (Dublin, Ireland) (MM ’25). Association for Computing Machinery, New York, NY, USA, 11677–11686. doi:10.1145/3746027.3755711 [12] Haoran Gao, Yuanhe Zhang, Zhenhong Zhou, Lei Jiang, Fanyu Meng, Yujia Xiao, Kun Wang, Yang Liu, and Junlan Feng. 2025. RECALLED: An Unbounded Resource Consumption Attack on Large Vision-Language Models. arXiv e-prints (2025), arXiv–2507. [13] Yichen Gong, Delong Ran, Jinyuan Liu, Conglei Wang, Tianshuo Cong, Anyu Wang, Sisi Duan, and Xiaoyun Wang. 2025. FigStep: jailbreaking large visionlanguage models via typographic visual prompts. In Proceedings of the ThirtyNinth AAAI Conference on Artificial Intelligence and Thirty-Seventh Conference on Innovative Applications of Artificial Intelligence and Fifteenth Symposium on Educational Advances in Artificial Intelligence (AAAI’25/IAAI’25/EAAI’25). AAAI Press, Article 2670, 9 pages. doi:10.1609/aaai.v39i22.34568 [14] Lukas Helff, Felix Friedrich, Manuel Brack, Patrick Schramowski, and Kristian Kersting. 2024. Llavaguard: Vlm-based safeguard for vision dataset curation and safety assessment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8322–8326. [15] Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. 2023. Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations. arXiv:2312.06674 [cs.CL] https://arxiv.org/abs/2312.06674 [16] Divyansh Kaushik, Douwe Kiela, Zachary C Lipton, and Wen-tau Yih. 2021. On the efficacy of adversarial data collection for question answering: Results from a large-scale randomized study. arXiv preprint arXiv:2106.00872 (2021). [17] Joonkyung Kim, Wenxi Chen, Davood Soleymanzadeh, Yi Ding, Xiangbo Gao, Zhengzhong Tu, Ruqi Zhang, Fan Fei, Sushant Veer, Yiwei Lyu, Minghui Zheng, and Yan Gu. 2026. Modular Safety Guardrails Are Necessary for FoundationModel-Enabled Robots in the Real World. arXiv:2602.04056 [eess.SY] https: //arxiv.org/abs/2602.04056

Shuo Shi et al.

[18] Youngwan Lee, Kangsan Kim, Kwanyong Park, Ilchae Jung, Soojin Jang, Seanie Lee, Yong-Ju Lee, and Sung Ju Hwang. [n. d.]. HoliSafe: Holistic Safety Benchmarking and Modeling for Vision-Language Model. ([n. d.]). [19] Lijun Li, Zhelun Shi, Xuhao Hu, Bowen Dong, Yiran Qin, Xihui Liu, Lu Sheng, and Jing Shao. 2025. T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generation. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 13381–13392. doi:10.1109/CVPR52734.2025.01249 [20] Yuyuan Li, Chaochao Chen, Yizhao Zhang, Weiming Liu, Lingjuan Lyu, Xiaolin Zheng, Dan Meng, and Jun Wang. 2023. Ultrare: Enhancing receraser for recommendation unlearning via error decomposition. Advances in Neural Information Processing Systems 36 (2023), 12611–12625. [21] Yuyuan Li, Yizhao Zhang, Weiming Liu, Xiaohua Feng, Zhongxuan Han, Chaochao Chen, and Chenggang Yan. 2025. Multi-Objective Unlearning in Recommender Systems via Preference Guided Pareto Exploration. IEEE Transactions on Services Computing (2025). [22] Zhaoyi Li, Xiaohan Zhao, Dong-Dong Wu, Jiacheng Cui, and Zhiqiang Shen. 2025. A frustratingly simple yet highly effective attack baseline: Over 90% success rate against the strong black-box models of gpt-4.5/4o/o1. arXiv preprint arXiv:2503.10635 (2025). [23] Qin Liu, Fei Wang, Chaowei Xiao, and Muhao Chen. 2025. VLM-Guard: Safeguarding Vision-Language Models via Fulfilling Safety Alignment Gap. arXiv preprint arXiv:2502.10486 (2025). [24] Liming Lu, Shuchao Pang, Siyuan Liang, Haotian Zhu, Xiyu Zeng, Aishan Liu, Yunhuai Liu, and Yongbin Zhou. 2025. Adversarial training for multimodal large language models against jailbreak attacks. arXiv preprint arXiv:2503.04833 (2025). [25] Haochen Luo, Jindong Gu, Fengyuan Liu, and Philip Torr. 2023. An image is worth 1000 lies: Transferability of adversarial images across prompts on vision-language models. In The Twelfth International Conference on Learning Representations. [26] OpenAI. 2026. Moderation API - OpenAI Platform Documentation. https:// platform.openai.com/docs/guides/moderation. Accessed: 2026-02-07. [27] Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Peter Henderson, Mengdi Wang, and Prateek Mittal. 2024. Visual adversarial examples jailbreak aligned large language models. In Proceedings of the AAAI conference on artificial intelligence, Vol. 38. 21527–21536. [28] Yiting Qu, Xinyue Shen, Yixin Wu, Michael Backes, Savvas Zannettou, and Yang Zhang. 2025. UnsafeBench: Benchmarking Image Safety Classifiers on Real-World and AI-Generated Images. In Proceedings of the 2025 ACM SIGSAC Conference on Computer and Communications Security (Taipei, Taiwan) (CCS ’25). Association for Computing Machinery, New York, NY, USA, 3221–3235. doi:10.1145/3719027.3765088 [29] Zedian Shao, Hongbin Liu, Yuepeng Hu, and Neil Zhenqiang Gong. 2024. Refusing Safe Prompts for Multi-modal Large Language Models. arXiv preprint arXiv:2407.09050 (2024). [30] Ilia Shumailov, Yiren Zhao, Daniel Bates, Nicolas Papernot, Robert Mullins, and Ross Anderson. 2021. Sponge examples: Energy-latency attacks on neural networks. In 2021 IEEE European symposium on security and privacy (EuroS&P). IEEE, 212–231. [31] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 8317–8326. [32] Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liangyan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, and Trevor Darrell. 2024. Aligning Large Multimodal Models with Factually Augmented RLHF. In Findings of the Association for Computational Linguistics: ACL 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 13088–13110. doi:10.18653/ v1/2024.findings-acl.775 [33] Simone Tedeschi, Felix Friedrich, Patrick Schramowski, Kristian Kersting, Roberto Navigli, Huu Nguyen, and Bo Li. 2024. ALERT: A Comprehensive Benchmark for Assessing Large Language Models’ Safety through Red Teaming. arXiv preprint arXiv:2404.08676 (2024). [34] Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, et al. 2023. DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models.. In NeurIPS. [35] Youze Wang, Wenbo Hu, Yinpeng Dong, Jing Liu, Hanwang Zhang, and Richang Hong. 2025. Align is not enough: Multimodal universal jailbreak attack against multimodal large language models. IEEE Transactions on Circuits and Systems for Video Technology (2025). [36] Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, PoSen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, Zac Kenton, Sasha Brown, Will Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hendricks, William Isaac, Sean Legassick, Geoffrey Irving, and Iason Gabriel. 2021. Ethical and social risks of harm from Language Models. arXiv:2112.04359 [cs.CL] https://arxiv.org/abs/2112.04359 [37] Fenghua Weng, Yue Xu, Chengyan Fu, and Wenjie Wang. 2025. Mmj-bench: A comprehensive study on jailbreak attacks and defenses for vision language

The Boy Who Cried Wolf: Adversarial Misclassification of Safe Inputs as Unsafe in Multimodal Guardrails

models. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. 27689–27697. [38] Peng Xie, Yequan Bie, Jianda Mao, Yangqiu Song, Yang Wang, Hao Chen, and Kani Chen. 2025. Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attacks. In Proceedings of the Computer Vision and Pattern Recognition Conference. 14679–14689. [39] Naen Xu, Jiayi Sheng, Changjiang Li, Chunyi Zhou, Yuyuan Li, Tianyu Du, Jun Wang, Zhihui Fu, Jinbao Li, and Shouling Ji. 2026. " I See What You Did There": Can Large Vision-Language Models Understand Multimodal Puns? arXiv preprint arXiv:2604.05930 (2026). [40] Naen Xu, Jinghuai Zhang, Ping He, Chunyi Zhou, Jun Wang, Zhihui Fu, Tianyu Du, Zhaoxiang Wang, and Shouling Ji. 2026. FraudShield: Knowledge Graph Empowered Defense for LLMs against Fraud Attacks. In Proceedings of the ACM Web Conference 2026. 2649–2660. [41] Naen Xu, Jinghuai Zhang, Changjiang Li, Hengyu An, Chunyi Zhou, Jun Wang, Boyu Xu, Yuyuan Li, Tianyu Du, and Shouling Ji. 2026. Bridging the Copyright Gap: Do Large Vision-Language Models Recognize and Respect Copyrighted Content?. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40. 35949–35957. [42] Chen Yeh, You-Ming Chang, Wei-Chen Chiu, and Ning Yu. 2024. T2Vs Meet VLMs: A Scalable Multimodal Dataset for Visual Harmfulness Recognition. Advances in Neural Information Processing Systems 37 (2024), 112950–112961. [43] Rui Yin, Tianxu Han, Naen Xu, Changjiang Li, Ping He, Chunyi Zhou, Jun Wang, Zhihui Fu, Tianyu Du, Jinbao Li, and Shouling Ji. 2026. Compiling Activation Steering into Weights via Null-Space Constraints for Stealthy Backdoors. arXiv:2604.12359 [cs.CR] https://arxiv.org/abs/2604.12359 [44] Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2024. A survey on multimodal large language models. National Science Review 11, 12 (Nov. 2024). doi:10.1093/nsr/nwae403 [45] Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, and Tat-Seng Chua. 2024. RLHFV: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 13807–13816. [46] Yi Zeng, Kevin Klyman, Andy Zhou, Yu Yang, Minzhou Pan, Ruoxi Jia, Dawn Song, Percy Liang, and Bo Li. 2024. Ai risk categorization decoded (air 2024): From government regulations to corporate policies. arXiv preprint arXiv:2406.17864 (2024). [47] Jiaming Zhang, Junhong Ye, Xingjun Ma, Yige Li, Yunfan Yang, Yunhao Chen, Jitao Sang, and Dit-Yan Yeung. 2025. AnyAttack: Towards Large-scale Self-supervised Adversarial Attacks on Vision-language Models. In Proceedings of the Computer Vision and Pattern Recognition Conference. 19900–19909. [48] Jiaming Zhang, Qi Yi, and Jitao Sang. 2022. Towards adversarial attack on vision-language pre-training models. In Proceedings of the 30th ACM International Conference on Multimedia. 5005–5013. [49] Qingzhao Zhang, Ziyang Xiong, and Z. Morley Mao. 2025. LLM Safeguard is a Double-Edged Sword: Exploiting False Positives for Denial-of-Service Attacks. arXiv:2410.02916 [cs.CR] https://arxiv.org/abs/2410.02916 [50] Rui Zhang, Zihan Wang, Tianli Yang, Hongwei Li, Wenbo Jiang, Qingchuan Zhao, Yang Liu, and Guowen Xu. 2025. Hidden tail: Adversarial image causing stealthy resource consumption in vision-language models. arXiv preprint arXiv:2508.18805 (2025). [51] Yi-Fan Zhang, Tao Yu, Haochen Tian, Chaoyou Fu, Peiyan Li, Jianshu Zeng, Wulin Xie, Yang Shi, Huanyu Zhang, Junkang Wu, et al. 2025. Mm-rlhf: The next step forward in multimodal llm alignment. arXiv preprint arXiv:2502.10391 (2025). [52] Xiangyu Zhao, Shengyuan Ding, Zicheng Zhang, Haian Huang, Maosongcao Maosongcao, Jiaqi Wang, Weiyun Wang, Xinyu Fang, Wenhai Wang, Guangtao Zhai, et al. 2025. Omnialign-v: Towards enhanced alignment of mllms with human preference. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 18490–18515. [53] Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongxuan Li, Ngai-Man Man Cheung, and Min Lin. 2023. On evaluating adversarial robustness of large visionlanguage models. Advances in Neural Information Processing Systems 36 (2023), 54111–54138. [54] Jiamu Zheng, Jinghuai Zhang, Tianyu Du, Xuhong Zhang, Jianwei Yin, and Tao Lin. 2025. CollabEdit: Towards Non-destructive Collaborative Knowledge Editing. arXiv:2410.09508 [cs.CL] https://arxiv.org/abs/2410.09508

Guard Model Safety Taxonomies

Table 6 and Table 7 present the safety taxonomies used by representative guard models in our evaluation. These taxonomies define the specific risk categories that guard models use to classify potentially harmful content.

KDD 2026, August 9–13, 2026, Jeju Island, Republic of Korea.