A Unified Evaluation Framework of Physical Adversarial Robustness in the Edge VLM Era

A Unified Evaluation Framework of Physical Adversarial Robustness in the Edge VLM Era

Eric Liang1

Alan Luo2,3∗ Kaiwen Yuan4 Saratoga High School 2 Google 3 University of Illinois Urbana-Champaign 4 Mecka.ai {eric.jinchen.liang@gmail.com, alanluo3@illinois.edu, kaiwen@mecka.ai}

arXiv:2610.29196v1 [cs.CV] 21 Oct 2026

1

Abstract Recent advancements in Vision-Language Models (VLMs) enable end-to-end visual grounding via language prompts and generative decoding, demonstrating largely improved zero-shot and open-vocabulary detection capabilities. As a result, VLMs are rapidly being deployed at the edge in safety-critical real-world systems. However, the physical adversarial robustness of edge VLMs has not been systematically studied, and the fundamental differences in output with modern object detectors create methodological challenges for fair evaluation. To compare the two families directly, we evaluate every selected representative model on one outcome: whether an adversarial patch worn on a person evades detection of the model. We report this by introducing a unified metric, conditional attack success rate (cASR), counted only on images the model still localizes under a matched gray occluder, and partition every failure into omission (no box), substitution (a box on the patch), and displacement (a box on neither). We benchmark the physical adversarial vulnerability of compact grounding VLMs against a range of object detectors, and find that the VLMs can be as vulnerable as the detectors. We disclose that failure mode is largely associated with the output format of a model rather than its architecture family. Of the three, only omission is visible to a downstream system; displacement and substitution return confident, well-formed boxes, making them potentially more dangerous failures. Further, we find that, under distribution shift, VLMs that primarily failed by omission instead shift to failing more by displacement/substitution. Thus, a grounding VLM can inherit the detector’s vulnerability while showing the possibility to adopt more dangerous failure modes. Our code and data are available at https: //github.com/ericliang1/physical-adversarial-robustness.

1

Introduction

Vision-language models (VLMs) are transitioning from describing a scene to localizing within one. Given a natural-language reference to an object, a VLM returns a bounding box. In embodied and autonomous settings [5, 29], a planner then conditions on that box directly [9], with no confidence score to threshold and no signal of uncertainty, so a correct box and an incorrect one arrive in the same form. Recent grounding VLMs are increasingly built for real-world deployment, with compact sub-4B variants suited to embedded accelerators [43, 30, 39, 4, 20]. Existing edge inference pipelines, however, were built around object detectors, and whether a grounding VLM is equally robust in their place cannot be settled on clean images, where both families localize a person at high accuracy. ∗ Corresponding author.

40th Conference on Neural Information Processing Systems (NeurIPS 2026). Workshop: VLM4RWD.

We therefore induce failures by training a physical adversarial patch against each model, a printed texture designed to suppress detection when worn on a person in the real world—an established threat to detectors that has not been tested against grounding VLMs on a common scale [33, 6]. The suppression objective of each adversarial patch drives down its corresponding model’s native output, making adversarial robustness directly comparable across architectures and exposing the inherent vulnerability of each model’s spatial grounding. We compare the two families across eleven models (Table 1) on the one predicate they share, whether a box lands on the target. We measure failure by introducing conditional attack success rate (cASR), which scores a patch only on images the model localizes under a matched gray occluder, and partition grounding failure into three modes—omission (no box), substitution (a box on the patch), and displacement (a box on neither); Figure 1 illustrates all three. We apply partitions in an occlusioncontrolled comparison of the two families across held-out and out-of-distribution images. Input

Output

Image with patch

Omission There is no person in the image.

Displacement

Vision Language Model

person[[40, 320, 380, 760]]

Instruction Please provide the bounding box coordinate of the region this sentence describes: <ref>person</ref>

Substitution person[[357, 380, 560, 583]]

Ground Truth Box

Predicted Box

Figure 1: The three failure modes: omission, displacement, and substitution. Two findings follow. First, grounding VLMs can be as vulnerable as the detectors under adversarial patches in deployment settings. Substituting one for the other does not necessarily reduce a system’s failure rate under a white-box attack. Second, the two families do not fail in the same way. Whether an attacked model omits or displaces is instead largely associated with its output format rather than its architecture family, while architecture also plays a role in the failure partition.

2

Related work

2.1

Object detection and grounding VLMs

Traditional object detectors [12, 27] have centered on convolutional neural networks (CNNs) for their superior capability for edge-device deployment due to low inference latency and excellent energy efficiency. The recent success of the transformer [34] has inspired vision transformer (ViT) based vision tasks [10, 14] that rethink the architecture in a natural language processing schema. Shortly after, transformer-based object detectors [7] came to dominate leaderboards [40, 44], demonstrating accuracy and generalization on large datasets. On the other hand, transformers still suffer from a lack of inductive biases and weaker performance on small and low-data problems [23, 10]. Therefore, both CNN and transformer models are commonly used for edge deployment. To make detectors more general purpose, open-vocabulary capabilities have been integrated into CNN [8] and transformer architectures [21], allowing detections to be driven by natural language prompts. More recently, general purpose large language models have trended towards multimodality and have thus also emerged into visual grounding capabilities [43, 30, 39, 32, 4]. These grounding models not only yield bounding box coordinates of objects but also query objects in an open-vocabulary manner. Even though most open-weight multimodal large language models (MLLMs) and VLMs are very heavy, with billions, and in the largest cases trillions, of parameters [32], there are also a handful of compact VLM variants that are promising for deployment in lightweight settings [39, 20]. As a result, it may be a matter of time before edge devices are armed with VLMs either in place or in 2

combination with detectors. Understanding the security of models in these two different detection schemas is therefore critical. 2.2

Adversarial attacks on object detection

The first adversarial attacks on modern object detectors date back to 2017, focusing on imperceptible perturbations applied to an image [38]. Later studies focused on transferability and generalization [33, 6] of adversarial attacks. These works utilized adversarial patches – textures capable of existing in diverse scenes in the real world, revealing a model’s vulnerabilities to specific localized patterns. The same idea has been further extended from the RGB domain to other (and mixed) modalities [2, 1]. Underneath the variety, the threat model is consistent: the attacker optimizes pixels in texture space against a model’s native output. However, language models admit no such formulation. Gradients do not act on discrete tokens, so attacks there are found by searching over candidate strings, with gradients through the token embeddings guiding the search [35, 45]. The form of an attack on a language interface is dramatically different: it can be typed, injected or retrieved. VLMs sit at the intersection of both methods and are exposed to both vulnerabilities. Early work showed that CLIP-style encoders can classify objects by text in the image rather than by the object itself [13], while later attacks introduced traditional imperceptible perturbation-based adversaries that transfer across models and prompts [42, 24]. A few recent works have begun to target grounded localization directly through training-time backdoors, though they are digital rather than physical [26, 17]. On the other hand, vision-language-action models, which use VLM backbones to act in robotics, have been demonstrated to fail by physical attacks in their visual input [36, 22]; however, these attacks target the model’s actions rather than its localization. Despite these advances, understanding the physical vulnerabilities of a model’s localization requires evaluating the way traditional detector attacks do [33]. Existing physical-attack evaluations cover detectors alone [18]; none of these works addresses real-time VLMs small enough to be deployed on edge devices, which our work aims to address.

3

Methodology

3.1

Models and datasets

For models, representative traditional closed-set object detectors [16, 15, 25], open-vocabulary detectors [8, 41, 21], and edge VLMs [43, 30, 39, 4, 20] whose parameter counts are no larger than 4B are selected for evaluation. We conduct our training on three datasets for people detection—COCO [19], PASCAL VOC [11] and AdvT-Shirt-1K [37]—and each contains ground-truth labels for every human in each scene. COCO and PASCAL VOC contain diverse person images in various scenes while AdvT-Shirt contains people with adversarial patches worn on shirts. However, we treat AdvT-Shirt the same as COCO and PASCAL VOC, as its garments were optimized against older detectors and have no effect on the models we test, as seen in the clean column of Table 1. We manually filter each dataset into subsets that contain only images with exactly one person. This ensures that every model is evaluated on a matched single-target detection task, removing ambiguity from VLM prompts, as VLMs generally output a single bounding box. Across all models and in-distribution datasets we use an 80/20 training/validation split, giving 1,324 training and 331 held-out validation images. We additionally evaluate out-of-distribution on 999 filtered single-person images from the Waymo Open Dataset [31], which represent street-level pedestrian views captured from a moving vehicle. Exact sources, filter counts, and split construction are given in Appendix B. For our experiments, all images are resized to the native input resolution of the vision model: all VLMs use 448 × 448 while all detectors use 640 × 640. 3.2

Adversarial patch training

We train on 1,324 images—622 COCO, 539 PASCAL VOC, 163 AdvT-Shirt—each containing exactly one person. For each image, a single adversarial patch is applied to the center of the person. The patch is a square texture with an original resolution of 160 × 160. When placed, it is resized 3

Clean Images

Detectors

Patch Applied

Closed-Set

Suppression Loss Open-Vocabulary

Corresponding Model

Adversarial Patches

Total Loss

EOT

Physical Constraints

…

Vision Language Model

Update Patch

Figure 2: Patch training pipeline.

Figure 3: Trained adversarial patches for each model. Patches trained on detectors contain high frequency patterns while patches trained on VLMs contain features that resemble text. relative to the person ground-truth label according to si = a · min(wi , hi ), where si is the side length of the placed patch in image i, wi and hi denote the width and height of the target bounding box, and a denotes the scaling factor, with a = 0.5 for our main experiment. This scale allows the patch to cover the center of the person while leaving important person features such as heads and limbs fully visible, representing the size of a feasibly wearable garment. We use white-box optimization to independently train an image-agnostic patch for each model, as summarized in Figure 2. Each attack is optimized to directly suppress the output of its respective target model. For detectors, patches are trained to suppress the final detection of the person. For VLMs, patches are trained to suppress the bounding box tokens, reducing the model’s likelihood of emitting the spatial information of the patched individual. The resulting patches are shown in Figure 3. Optimization is performed across 500 epochs with a learning rate of 0.01 for all models, allowing all patches to converge and ensuring a fair comparison between them. To make the patches physically robust and effective across different scenes, we train with Expectation Over Transformation (EOT) [3], applying random rotations drawn uniformly from [−20◦ , 20◦ ] and multiplicative brightness jitter drawn uniformly from [0.9, 1.1] (i.e., ±10%). The patch scale is applied when the patch is placed on the target and is not part of the EOT distribution. All patches are applied digitally; “physical” represents the threat model rather than the evaluation medium. The patch is trained using the loss function L = Ls + αLnps + βLtv ,

(1)

where Ls suppresses the output, Lnps enforces printability [28], Ltv enforces smoothness, and α and β are two hyperparameters to balance the scale. We set α = 0.01 and β = 0.01, keeping the 4

printability and smoothness terms small enough to constrain the patch to be physically realizable without trading away attack strength. Adapted from Thys et al. [33], the non-printability and smoothness terms are 1 X min (p − c)2 , c∈C |P |

(2)

1 X (|pi,j − pi+1,j | + |pi,j − pi,j+1 |) , N i,j

(3)

Lnps =

p∈P

Ltv =

where pi,j denotes a patch pixel, p a pixel of the patch P , c a color in the set of printable colors C, and |P | the number of patch pixels; in Ltv , each absolute difference is taken per color channel (red, green, blue) and N is the total number of differences. For each model, Ltv and Lnps remain the same, and are used to constrain the patch to be physically realizable, while Ls performs the same suppression task on the final differentiable prediction scores before the model’s final output decision, which is unique to each model’s architecture and native output structure. Concretely, Ls takes one of three forms,  closed-set detector,  maxk zk Ls = maxk sim(rk , t) open-vocabulary detector, (4)   log pθ (b | q) vision–language model, where zk is the person confidence of proposal k; sim(rk , t) is the similarity between region k and the “person” text embedding t; and pθ (b | q) is the likelihood of decoding the target’s box tokens b under prompt q. Minimizing Ls suppresses the person confidence, the region–text similarity, or the grounding likelihood, respectively.

4

Evaluation framework

We create a unified evaluation framework that aligns adversarial attack evaluation across heterogeneous model architectures. The framework ensures that the effects of adversarial attacks can be directly compared by framing success as the ability of a patched target individual to evade detection. 4.1

Configuration

The 331 validation images comprise 155 COCO, 135 PASCAL VOC, and 41 AdvT-Shirt; the 999 Waymo images are evaluated separately as the out-of-distribution set. Unless stated otherwise, the following default conditions hold: the overlap threshold is τ = 0.5, the standard detection criterion [11]; the patch is placed at the scale it was trained at; and detectors are read at their canonical confidence threshold. 4.2

Output alignment

To align with detectors, we prompt each VLM to produce one bounding box per image in its native grounding format, then normalize all outputs to a standard xyxy format. Any output that is malformed or fails to correspond to a valid bounding box is treated as a detection failure, enforcing evaluation based on spatial correctness. Each model is scored on one box b̂i : for a detector, it is the highest-confidence person detection clearing γ; for a VLM, it is the single box it emits. Since traditional object detectors produce many detections per target instance ranked by confidence scores, while vision-language models produce a single detection, a detector is read at a chosen confidence threshold whereas a VLM is read at its one emission. For VLMs, we decode greedily and do not consider the temperature, which is standard for single-target detection tasks. For detectors, we evaluate at the confidence thresholds at which their family is conventionally operated, ensuring that all models are evaluated in deployment-relevant settings. Closed-set detectors are read at the confidence threshold γ = 0.5. Open-vocabulary detectors are read at a lower confidence threshold, at γ = 0.3. 5

4.3

Occlusion control and conditional ASR

In evaluation, a gray digital patch can cause a failure by occluding part of the target, causing a detection to fail. As a result, standard attack success rate (ASR) can overestimate the adversarial effect of a patch, while subtraction-based flooring methods may underestimate the adversarial effect when the gray patch itself causes a high failure rate. To separate true adversarial success from occlusion and baseline model error, we evaluate each adversarial patch only on the images for which the corresponding model can successfully detect the person under the gray-patched condition. We define this metric as conditional ASR (cASR). Let G be the set of images the model localizes correctly (Intersection over Union, IoU ≥ τ , against the ground-truth box bi ) under the gray occluder, and let fi = 1 when IoU(b̂i , bi ) < τ , where b̂i is the box predicted on the adversarially patched image. Then cASR =

1 X fi , |G|

(5)

i∈G

the fraction of gray survivors that the adversarial patch breaks. The floor’s magnitude corresponds to localization tightness: tight boxes lose overlap once the target center is covered, whereas loose ones retain enough to clear τ . The models most affected by the control are therefore the most precise rather than the weakest—and a benchmark reporting raw attack success would penalize them for that precision. 4.4

Failure partition

Conditional attack success quantifies failure frequency but not failure mode. Omission is tracked when no box of a proper format forms. Substitution is tracked when a box is formed and closely aligns with the patch, measured by IoU(b̂i , Fi ) ≥ 0.5, where Fi is the patch footprint. Displacement is tracked when a box is formed but lands on neither the target nor the patch, falling below τ against bi and below 0.5 against Fi . The three are exclusive by construction and exhaustive by the last case, so they sum to cASR.

5

Results

Baseline localization. Table 1 reports localization with no patch applied, where a model counts as correct if it emits a box overlapping the ground truth at IoU ≥ 0.5. The criterion is identical for every model. In-distribution accuracy spans 83.0% to 100.0%. On the out-of-distribution Waymo set accuracy drops for every model, reflecting the harder detection task of street-level pedestrian views from a moving vehicle. Most models remain well above 65%, while MiniCPM-V 4.6 falls to 51.1% and InternVL3-2B to 27.6% on the same images. By construction, cASR counts a failure only on images the model localizes correctly under the matched gray occluder (Eq. 5), so no failure below is attributable to baseline localization. This conditioning means each model’s Waymo cASR is read over its own, smaller gray-control set. For InternVL3-2B and MiniCPM-V 4.6, which localize driving pedestrians poorly, this set is as small as 305 and 366 of 999 images, respectively. 5.1

VLMs can be as vulnerable as detectors

Table 2 reports cASR (Eq. 5), computed over the gray-control set, so occlusion alone contributes near zero. Grounding VLMs rank among the most affected models in the benchmark: InternVL3-2B and PaliGemma 2-3B fail on 57.7% and 55.2% of the gray-control set, higher than two of the six detectors we test. AdvT-Shirt gives the cleanest comparison, as every model localizes at 95% or above with no patch applied. With the adversarial patch applied, InternVL3-2B and PaliGemma 2-3B fail on 100% and 97.6% of their gray-control sets, comparable to the worst detectors (Table 1), YOLO11s and YOLO26s, which both reach 100%. Substituting a VLM for a detector therefore does not necessarily lower the failure rate. The most robust models also come from both families. MiniCPM-V 4.6 is the most robust overall, failing on only 9.2% of the gray-control set. YOLO-World, though a detector, sits at 18.2%, more robust than three of the five VLMs. The VLMs span 9.2% to 57.7%, while the detectors span 18.2% to 78.3%. The two ranges largely overlap, with the variation within each family exceeding the 6

Table 1: Clean detection accuracy and conditional attack success (cASR). Bold marks the lowest cASR within each family. In-distribution (trained)

Out-of-distribution

VOC

Waymo

COCO Model

AdvT-Shirt

Params

Clean

cASR

Clean

cASR

Clean

cASR

Clean

cASR

Closed-set detectors YOLO11s 9.5 M YOLO26s 9.5 M RT-DETRv2 20.2 M

96.1 95.5 94.2

80.3 66.1 63.0

91.9 92.6 92.6

66.7 50.5 53.8

100.0 100.0 100.0

100.0 100.0 75.6

74.5 75.5 65.8

86.1 38.7 80.4

Open-vocabulary detectors YOLO-World 76.2 M 96.1 GDINO-Tiny 172.3 M 97.4 GDINO-Base 232.8 M 100.0

25.6 64.2 60.0

97.8 97.8 97.8

15.0 56.1 47.7

100.0 100.0 100.0

2.5 85.4 58.5

75.2 89.8 90.7

25.3 62.1 40.2

Vision-language models MiniCPM-V 4.6 1.3 B InternVL3-2B 2.1 B Qwen3-VL-2B 2.1 B PaliGemma 2-3B 3.0 B LFM2.5-VL-3B 3.1 B

10.1 59.2 22.2 59.3 16.4

83.0 91.1 94.8 94.1 91.1

10.3 41.5 22.6 37.6 11.4

95.1 97.6 100.0 100.0 100.0

2.5 100.0 7.5 97.6 14.6

51.1 27.6 90.3 86.7 87.8

9.6 35.4 39.7 15.4 5.8

94.8 98.7 99.4 97.4 96.1

Table 2: Failure partition into omission (O), substitution (S), and displacement (D). In-distribution Model

Grammar

Waymo (out-of-distribution)

cASR

O

S

D

cASR

O

S

D

Closed-set detectors YOLO11s score-gated YOLO26s score-gated RT-DETRv2 score-gated

78.3 65.3 61.9

72.8 62.2 53.5

0.0 0.0 0.0

5.5 3.1 8.4

86.1 38.7 80.4

81.5 31.0 61.2

0.0 0.0 0.5

4.6 7.6 18.7

Open-vocabulary detectors YOLO-World text-match GDINO-Tiny text-match GDINO-Base text-match

18.2 63.6 54.9

15.6 49.4 29.9

0.7 3.4 10.7

1.8 10.8 14.3

25.3 62.1 40.2

22.4 8.8 1.4

0.3 16.6 18.0

2.6 36.7 20.9

Vision-language models MiniCPM-V 4.6 free-text InternVL3-2B gated Qwen3-VL-2B free-text PaliGemma 2-3B gated LFM2.5-VL-3B free-text

9.2 57.7 20.1 55.2 14.3

0.7 56.7 0.0 53.4 0.0

0.3 0.0 1.5 0.0 5.2

8.1 1.0 18.7 1.9 9.1

9.6 35.4 39.7 15.4 5.8

0.0 4.6 0.0 8.9 0.0

0.3 1.3 0.9 0.1 1.1

9.3 29.5 38.8 6.3 4.7

difference between them. Robustness is therefore largely a property of the individual model rather than its family. The difference between the most and least affected models therefore reflects what the attack can suppress. The attack drives some quantity down (a detector’s confidence, a VLM’s token likelihood), which removes the box only when a single token gates the box’s existence (Eq. 4). Producing no box guarantees a failure, whereas perturbing coordinates only constitutes a failure if the box moves far enough to fall below τ . InternVL3-2B and PaliGemma 2-3B open their output with such a gating token, so driving down its likelihood produces an omission; they are the two most affected VLMs we test. A free-text model has no such token. It writes coordinates as ordinary text that the decoder produces whether or not the target is visible, so the attack cannot delete the box, only mislocalize it. The free-text VLMs (LFM2.5-VL-3B, MiniCPM-V 4.6, and Qwen3-VL-2B) are also the three least affected as measured by cASR. 5.2

Failure modes

Table 2 partitions each model’s failures into omission, substitution, and displacement (Section 4.4). 7

Omission and displacement. The two VLM grammars occupy opposite extremes of the failure mode partition. The gated VLMs, PaliGemma 2-3B and InternVL3-2B, fail almost entirely by omission: 96.7% and 98.3% of their failures produce no box, a higher share than any detector we test. The score-gated detectors behave the same way, with the YOLO models failing by omission on 93% to 95% of failures. At the other end, the free-text VLMs Qwen3-VL-2B and LFM2.5-VL-3B record exactly zero omissions at every threshold we sweep, and MiniCPM-V 4.6 fewer than 1%. What these models produce instead is displacement, a box that parses correctly but lands on neither the person nor the patch. Notably, Qwen3-VL-2B and LFM2.5-VL-3B share a grammar yet fail at different rates and split differently between displacement and substitution (displacement accounts for 18.7% of Qwen3-VL-2B’s 20.1% and 9.1% of LFM2.5-VL-3B’s 14.3%). Therefore, under a shared suppression objective that lowers the likelihood of the target box without targeting an empty response, output format is associated with whether a box is emitted, but not how often or where it lands. Alternative prompts can still shift this balance, as MiniCPM-V 4.6 omits on 7–9% of attacked images under alternative phrasings (Table 3). Because grammar fixes emission but not placement, a stricter overlap criterion should affect only the models that emit a box. Raising the IoU threshold (τ ) from 0.5 to 0.9 (Figure 5 in Appendix A), the free-text models’ failure rates climb sharply while the gated models’ barely move. Qwen3-VL2B’s cASR reaches 95.6%—the largest aggregated in-distribution cASR any model reaches in the benchmark—while its omission holds at exactly zero throughout, the entire increase being fully attributable to boxes it still emits. PaliGemma 2-3B and InternVL3-2B only move from 55.2% to 68.6% and 57.7% to 62.8%, remaining omission-dominated at every point. Thus raising τ does not affect omissions: with no box emitted, they fail at every threshold regardless. It instead converts displaced boxes into failures as they stop clearing the threshold. Substitution. A box that lands on the patch itself rather than the person similarly occurs in models whose emission is not gated on the target’s presence. Every closed-set detector and both gated VLMs (PaliGemma 2-3B, InternVL3-2B) register exactly zero. Every other model registers a nonzero rate: the region-text matchers (GDINO-Base 10.7%, GDINO-Tiny 3.4%, YOLO-World 0.7%) and the free-text VLMs (LFM2.5-VL-3B 5.2%, Qwen3-VL-2B 1.5%, MiniCPM-V 4.6 0.3%). Substitution cuts across architectures and arises in the models where neither a confidence threshold nor a gating token withholds the box. In deployment, substitution and displacement carry worse consequences than omission because both emit a box. A downstream system can detect an omission, since a model that returns nothing signals its own failure. Substitution and displacement, on the other hand, deliver a confident, well-formed box at an incorrect location, which may be indistinguishable from a correct detection. Substitution is arguably the more severe of the two. It inverts the attack, supplying a stronger signal that the model reports as the person rather than suppressing the person’s own, so the system’s attention is anchored to an object the attacker carries rather than to empty space. Its box also takes the patch’s size, no larger than the person’s at every scale we evaluate, so any spatial feature a system infers from the bounding box (such as distance) is systematically distorted. A single success rate from prior work collapses these distinctions, directly motivating our proposed failure partition. 5.3

Under distribution shift

The Waymo set holds every model to the same task on a domain none of the patches were trained against, approximating the distribution shift of deployment. cASR moves in both directions, so the distribution shift is not a uniform increase in difficulty. What changes is the failure mode. Indistribution, InternVL3-2B, GDINO-Base, and GDINO-Tiny fail predominantly by omission. On the Waymo set, their omission rate drops sharply (InternVL3-2B from 56.7% to 4.6%, GDINO-Base from 29.9% to 1.4%, and GDINO-Tiny from 49.4% to 8.8%) while their overall failure rates fall much less in comparison. Their remaining failures are instead registering as displacement or substitution, the modes a downstream system cannot detect. InternVL3-2B’s displacement rises from 1.0% to 29.5%, GDINO-Tiny’s substitution from 3.4% to 16.6% and its displacement from 10.8% to 36.7%, and GDINO-Base’s substitution from 10.7% to 18.0%. Counterintuitively, InternVL3-2B’s cASR decreases (57.7% to 35.4%), yet most of its remaining failures are now undetectable. The closed-set detectors exhibit this shift far less: YOLO11s, YOLO26s, and RT-DETRv2 all remain omission-dominated on Waymo. In contrast, the open-vocabulary detectors and grounding VLMs 8

do lose detectability. The one exception is YOLO-World, the most robust among them, which stays omission-dominated throughout. Further, gated VLMs lose a key safety property under the distribution shift: in-distribution, InternVL32B and PaliGemma 2-3B fail by omission on 98.3% and 96.7% of their failures. On Waymo, PaliGemma 2-3B’s omission falls from 53.4% to 8.9% and InternVL3-2B’s nearly vanishes. Among the grounding VLMs, these two models are the only ones that predominantly fail visibly by omission, yet even that largely disappears under the distribution shift—highlighting that grounding VLMs can be as vulnerable as, or more vulnerable than, detectors in deployment settings.

6

Limitations

Our adversarial attacks are digitally simulated, and are not evaluated in physical settings. The patches are optimized to be printable and pose-robust, which is what “physical” refers to throughout; no printed patch or camera capture was evaluated, so we do not measure physical-world transfer. In addition, we only utilize white-box attack methods, which assume full access to the model and do not reflect real-world deployment scenarios. Future work should focus on evaluating the adversarial effect of physically printed patches and test black-box methods. Furthermore, we test only a suppression objective, and future work should evaluate persistence of the failure-mode partitions under different attack objectives.

7

Conclusion

Under one attack framework, one occlusion control, and a single shared predicate, edge grounding VLMs can be as vulnerable as the classic detectors, yet they do not fail in the same way: the form of each failure is largely associated with the model’s output format rather than its architecture family. This difference carries the deployment consequence. A patch trained to suppress the model’s detection of the person produces no box in some models and a well-formed, incorrectly placed box in others. Only the first is visible to a downstream system, while the second may be indistinguishable from a correct detection. Any grounding model bound for deployment should therefore be evaluated on two axes: how often it fails under physical attack, and whether those failures are visible in its output. The first requires an occlusion-controlled measurement, which our benchmark provides. Therefore, as evidenced by our controlled comparison, the shift from detectors to grounding VLMs does not reduce a system’s vulnerability to physical attack. Under the distribution shift of real-world deployment, it may render the system even more vulnerable—failing in modes that are invisible to the downstream systems relying on it.

Acknowledgments and Disclosure of Funding We thank our reviewers for helpful discussions. This work received no external funding.

References [1] Mazen Abdelfattah, Kaiwen Yuan, Z Jane Wang, and Rabab Ward. Adversarial attacks on camera-lidar models for 3d car detection. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2189–2194. IEEE, 2021. [2] Mazen Abdelfattah, Kaiwen Yuan, Z Jane Wang, and Rabab Ward. Towards universal physical attacks on cascaded camera-lidar 3d object detection models. In 2021 IEEE international conference on image processing (ICIP), pages 3592–3596. IEEE, 2021. [3] Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adversarial examples. In International conference on machine learning, pages 284–293. PMLR, 2018. [4] Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631, 2025. 9

[5] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-languageaction models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023. [6] Tom B. Brown, Dandelion Mané, Aurko Roy, Martín Abadi, and Justin Gilmer. Adversarial patch. arXiv preprint arXiv:1712.09665, 2017. [7] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213–229. Springer, 2020. [8] Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, and Ying Shan. Yolo-world: Real-time open-vocabulary object detection. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16901–16911. IEEE, 2024. [9] Thierry Deruyttere, Simon Vandenhende, Dusan Grujicic, Luc Van Gool, and Marie-Francine Moens. Talk2Car: Taking control of your self-driving car. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2088–2098. Association for Computational Linguistics, 2019. [10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. [11] Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman. The PASCAL visual object classes (VOC) challenge. International Journal of Computer Vision, 88(2):303–338, 2010. doi: 10.1007/s11263-009-0275-4. [12] Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 1440–1448, 2015. [13] Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill, 2021. doi: 10.23915/distill.00030. [14] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In 2022 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 15979–15988. IEEE, 2022. [15] Glenn Jocher and Jing Qiu. Ultralytics YOLO26: Unified real-time end-to-end vision models. arXiv preprint arXiv:2606.03748, 2026. [16] Rahima Khanam and Muhammad Hussain. Yolov11: An overview of the key architectural enhancements. arXiv preprint arXiv:2410.17725, 2024. [17] Junxian Li, Beining Xu, Simin Chen, Jiatong Li, Jingdi Lei, Haodong Zhao, and Di Zhang. Iag: Input-aware backdoor attack on vlm-based visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27872–27883, 2026. [18] Jiawei Lian, Jianhong Pan, Lefan Wang, Yi Wang, Lap-Pui Chau, and Shaohui Mei. Padetbench: Towards benchmarking physical attacks against object detection. arXiv preprint arXiv:2408.09181, 2024. [19] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In Computer Vision – ECCV 2014, volume 8693 of Lecture Notes in Computer Science, pages 740–755. Springer, 2014. doi: 10.1007/978-3-319-10602-1_48. [20] Liquid AI. LFM2.5-VL-3B. Model card, Hugging Face, 2026. https://huggingface.co/ LiquidAI/LFM2.5-VL-3B. 10

[21] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European conference on computer vision, pages 38–55. Springer, 2024. [22] Hui Lu, Yi Yu, Yiming Yang, Chenyu Yi, Qixin Zhang, Bingquan Shen, Alex C Kot, and Xudong Jiang. When robots obey the patch: Universal transferable patch attacks on visionlanguage-action models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22867–22878, 2026. [23] Alan Luo and Kaiwen Yuan. Simple self-organizing map with vision transformers. IEEE Signal Processing Letters, 33:331–335, 2026. doi: 10.1109/LSP.2025.3643388. [24] Haochen Luo, Jindong Gu, Fengyuan Liu, and Philip Torr. An image is worth 1000 lies: Transferability of adversarial images across prompts on vision-language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview. net/forum?id=nc5GgFAvtk. [25] Wenyu Lv, Yian Zhao, Qinyao Chang, Kui Huang, Guanzhong Wang, and Yi Liu. RT-DETRv2: Improved baseline with bag-of-freebies for real-time detection transformer. arXiv preprint arXiv:2407.17140, 2024. [26] Ankita Raj and Chetan Arora. Backdoor attacks on open vocabulary object detectors via multi-modal prompt tuning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 37849–37857, 2026. [27] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779–788, 2016. [28] Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K. Reiter. Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 1528–1540. ACM, 2016. doi: 10.1145/2976749.2978392. [29] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. CLIPort: What and where pathways for robotic manipulation. In Proceedings of the 5th Conference on Robot Learning (CoRL), volume 164 of Proceedings of Machine Learning Research, pages 894–906. PMLR, 2021. [30] Andreas Steiner, André Susano Pinto, Michael Tschannen, Daniel Keysers, Xiao Wang, Yonatan Bitton, Alexey Gritsenko, Matthias Minderer, Anthony Sherbondy, Shangbang Long, et al. Paligemma 2: A family of versatile vlms for transfer. arXiv preprint arXiv:2412.03555, 2024. [31] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2443–2451. IEEE, 2020. [32] Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, Jianfeng Cai, Xinyuan Cai, Peizhou Cao, Yuxuan Cao, Ziwei Chai, Y Charles, et al. Kimi k3: Open frontier intelligence. arXiv preprint arXiv:2607.24653, 2026. [33] Simen Thys, Wiebe Van Ranst, and Toon Goedemé. Fooling automated surveillance cameras: adversarial patches to attack person detection. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 49–55. IEEE, 2019. [34] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. [35] Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. Universal adversarial triggers for attacking and analyzing nlp. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 2153–2162, 2019. 11

[36] Taowen Wang, Cheng Han, James Liang, Wenhao Yang, Dongfang Liu, Luna Xinyu Zhang, Qifan Wang, Jiebo Luo, and Ruixiang Tang. Exploring the adversarial vulnerabilities of visionlanguage-action models in robotics. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pages 6948–6958. IEEE, 2025. [37] Zhixiang Wang, Xiaosen Wang, Bo Wang, Siheng Chen, Zhibo Wang, Xingjun Ma, and YuGang Jiang. Diffpatch: Generating customizable adversarial patches using diffusion models. arXiv preprint arXiv:2412.01440, 2024. [38] Cihang Xie, Jianyu Wang, Zhishuai Zhang, Yuyin Zhou, Lingxi Xie, and Alan Yuille. Adversarial examples for semantic segmentation and object detection. In 2017 IEEE international conference on computer vision (ICCV), pages 1378–1387. IEEE, 2017. [39] Tianyu Yu, Zefan Wang, Chongyi Wang, Fuwei Huang, Wenshuo Ma, Zhihui He, Tianchi Cai, Weize Chen, Yuxiang Huang, Ranchi Zhao, et al. Minicpm-v 4.5: Cooking efficient mllms via architecture, data, and training recipe. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11704–11715, 2026. [40] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and HeungYeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint arXiv:2203.03605, 2022. [41] Xiangyu Zhao, Yicheng Chen, Shilin Xu, Xiangtai Li, Xinjiang Wang, Yining Li, and Haian Huang. An open and comprehensive pipeline for unified object grounding and detection. arXiv preprint arXiv:2401.02361, 2024. [42] Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongxuan Li, Ngai-Man Cheung, and Min Lin. On evaluating adversarial robustness of large vision-language models. In Advances in Neural Information Processing Systems (NeurIPS), 2023. [43] Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479, 2025. [44] Zhuofan Zong, Guanglu Song, and Yu Liu. Detrs with collaborative hybrid assignments training. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 6725–6735. IEEE, 2023. [45] Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023.

12

A

Ablation studies

We report four ablations, each varying a single parameter of the attack while holding the rest of the pipeline constant: patch scale, overlap threshold, VLM prompt, and detector confidence threshold. All four are evaluated on the 331 validation images at the canonical settings, conditioned on the gray-control set. A.1

Patch size ablation

Figure 4 sweeps the patch scale a ∈ {0.25, 0.5, 0.75, 1.0}. cASR generally rises with scale, though the increase is not monotonic. The ordering of vulnerability is broadly preserved across the sweep, meaning the adversarial patches are robust to scale and reflect vulnerability rather than the choice of that parameter. 100

cASR (%)

80 60 40 20 0

0.25

0.5

0.75

1

YOLO11s YOLO26s RT-DETRv2 YOLO-World GDINO-Tiny GDINO-Base InternVL3-2B PaliGemma 2-3B Qwen3-VL-2B MiniCPM-V 4.6 LFM2.5-VL-3B

Patch scale a

Figure 4: Conditional attack success versus patch scale. A.2

IoU threshold ablation

Figure 5 sweeps the overlap threshold τ from 0.5 to 0.9. The size of the increase separates models by failure mode rather than family. Omission-dominated models show comparatively little change, because a box that is never emitted fails at every threshold. Models that always emit a box accumulate failures as the requirement tightens, with Qwen3-VL-2B rising from 20.1% to 95.6%. 100

cASR (%)

80 60 40 20 0

0.5

0.6

0.7

0.8

0.9

YOLO11s YOLO26s RT-DETRv2 YOLO-World GDINO-Tiny GDINO-Base InternVL3-2B PaliGemma 2-3B Qwen3-VL-2B MiniCPM-V 4.6 LFM2.5-VL-3B

IoU threshold τ

Figure 5: Conditional attack success versus IoU threshold. A.3

VLM prompt ablation

Table 3 shows prompt sensitivity, re-evaluating each VLM patch under its training prompt and three paraphrases with the patch itself held fixed. The failure rate varies by prompt, so a single prompt should not be read as a model’s robustness. However, the attacks mostly still generalize across various 13

different prompts, emphasizing deployment vulnerability. The failure partition remains stable: the gated grammars remain omission-dominated under all four prompts, and Qwen3-VL-2B records exactly zero omissions under every prompt we test.

Table 3: Conditional attack success and failure partition across prompts. Prompt

cASR Omission Substitution Displacement

PaliGemma 2-3B (gated) detect all people (TRAINED) detect all pedestrians detect all humans detect all persons

55.2 57.0 43.4 39.2

53.4 55.0 39.7 38.0

0.0 0.0 0.0 0.0

1.9 2.0 3.7 1.2

57.7

56.7

0.0

1.0

29.0

27.4

0.0

1.7

45.9

39.3

0.0

6.7

28.0

25.2

0.0

2.8

0.7

0.3

8.1

7.5

0.4

11.8

9.3

0.0

33.3

7.3

1.2

21.6

20.1

0.0

1.5

18.7

17.2

0.0

1.1

16.1

17.1

0.0

3.5

13.6

17.9

0.0

1.5

16.3

14.3

0.0

5.2

9.1

1.6

0.0

0.0

1.6

2.3

0.3

0.0

2.0

1.3

0.0

0.0

1.3

InternVL3-2B (gated) Please provide the bounding box coordinate of the region this sentence describes: <ref>person</ref> (TRAINED) Please detect the person and return the bounding box. Where is the person? Give the bounding box coordinates. Locate the person and provide the bounding box coordinates.

MiniCPM-V 4.6 (free-text) Locate the person. Output ONLY the 9.2 bounding box [x1, y1, x2, y2] in 0–1000 normalized coordinates, nothing else. (TRAINED) Detect the person and output the bounding 19.7 box [x1, y1, x2, y2] in 0–1000 coordinates. Identify the person and give the bounding 42.7 box [x1, y1, x2, y2] in 0–1000 coordinates. Find the person and output the bounding 30.1 box [x1, y1, x2, y2] in 0–1000 coordinates. Qwen3-VL-2B (free-text) Locate every person in the image and output the bounding box coordinates as JSON in the form {“bbox_2d”: [x1, y1, x2, y2], “label”: “person”}. (TRAINED) Locate the person and return {“bbox_2d”: [x1, y1, x2, y2], “label”: “person”}. Where is the person? Output {“bbox_2d”: [x1, y1, x2, y2], “label”: “person”}. Detect the person and output the bounding box as JSON {“bbox_2d”: [x1, y1, x2, y2], “label”: “person”}. LFM2.5-VL-3B (free-text) Please provide the bounding box coordinate of the region this sentence describes: the person. (TRAINED) Locate the person in the image and output the bounding box. Detect the person and output the bounding box [x1, y1, x2, y2]. Where is the person? Give the bounding box [x1, y1, x2, y2].

14

A.4

Detector confidence threshold ablation

Table 4 varies the confidence threshold γ at which each detector is read. As the confidence requirement increases, attack success increases for every model, suggesting that tasks that require more precision become more vulnerable. Table 4: Conditional attack success versus detector confidence threshold.

B

Detector

cASR (%)

Closed-set detectors γ 0.1

0.25

0.5

0.7

YOLO11s YOLO26s RT-DETRv2

73.1 56.7 49.8

78.3 65.3 61.9

83.8 67.9 68.6

Open-vocabulary detectors γ 0.1 0.2

0.3

0.4

YOLO-World GDINO-Tiny GDINO-Base

18.2 63.6 54.9

23.1 81.2 77.4

67.2 40.7 34.7

11.1 4.5 3.6

16.8 29.3 22.5

Dataset construction

Table 5 reports the filtering counts for each source dataset. From COCO we only take val2017, to prevent data leakage as some models are trained on train2017; from PASCAL VOC, the val sets of 2007 and 2012; from AdvT-Shirt, its 1-person folder; and from Waymo, the training split of v2 perception. Each dataset was first filtered to one-person images using its ground-truth bounding box labels. We then manually removed those containing extra unannotated people, and dropped Waymo’s night-time images to ensure proper visibility. Figure 6 shows example images from the final version of each dataset. Table 5: Per-dataset filtering counts. Source dataset COCO val2017 PASCAL VOC 2007+2012 val AdvT-Shirt-1K, 1-person subset Waymo v2 perception, training split

Original

One-person pool

Kept

Kept (%)

5,000 8,333 208 4,662

925 987 208 1,014

777 674 204 999

15.5 8.1 98.1 21.4

15

COCO PASCAL VOC AdvTshirt Waymo

Figure 6: Example images from each source dataset.

16

Related documents

• A Unified Evaluation Framework of Physical Adversarial Robustness in the Edge VLM Era#1413801
• Black-Box Adversarial Patch Attacks on VLAs via Ancestor VLM Exploitation#1234075
• Black-Box Adversarial Patch Attacks on VLAs via Ancestor VLM Exploitation#1227262
• Speed in the Blind Spot: An Interpretability Analysis of Dynamic Perception in VLMs for Autonomous Driving#1132247
• VLM4Cluster: Benchmarking Deep Clustering In the Era of Vision-Language Pre-training#1131849