Detect and Suppress: A Mechanistic Defense against Adversarial Patches in VLA Models

Detect and Suppress: A Mechanistic Defense against Adversarial Patches in VLA Models Yukiya Horiba Keio University

Koshiro Aoki Waseda University

Shunsuke Yasuki The University of Tokyo

Bum Jun Kim The University of Tokyo

arXiv:2610.03498v1 [cs.RO] 2 Oct 2026

Taiki Miyanishi The University of Tokyo

a Adversarial attacks “Put the black bowl in the drawer and close it.”

b Diagnosis using SAE

c Online intervention

Train VLA

Adversarial patch t=0

❌ Fail to follow

VLA $ Normal

Under attack

VLA Detection & suppression

Benign trajectory t=40

# Adversarially affected

Sparse Autoencoder (SAE)

✅ Successfully follow

Figure 1: Overview of our SAE-based defense. Abstract: Adversarial patches can disrupt Vision-Language-Action (VLA) models by manipulating visual observations, leading to failures in robot control. However, it remains poorly understood which internal mechanisms underlie these failures and how targeted interventions can mitigate them. In this work, we mechanistically analyze VLA representations using a sparse autoencoder (SAE) and identify a feature whose activation strongly correlates with the presence of an adversarial patch. Based on this analysis, we suppress the identified feature at inference time only when a linear probe detects an attack. This intervention improves robustness without the cost of fine-tuning the VLA. We evaluate our method against VLA adversarial patch attacks on LIBERO-10. Conditional intervention improves success rate under intermittent attacks, whereas continuously applying the same intervention substantially degrades policy performance. These results show that attack-related internal representations can provide useful targets for VLA adversarial defense and that controlling when to intervene is important for limiting disruption to nominal policy behavior. Keywords: Vision-Language-Action Models, Adversarial Robustness, Mechanistic Interpretability

1

Introduction

Adversarial patches can disrupt vision-language-action (VLA) policies by altering the visual observations used for robot control [1, 2]. Existing VLA defenses modify either the inputs, through image preprocessing [3], or the model parameters, through adversarial fine-tuning [2]. Adversarial finetuning, however, incurs the computational cost of additional policy training. We ask whether VLA

robustness can instead be improved by directly targeting attack-related internal features at inference time, without fine-tuning the policy. Mechanistic interpretability provides a basis for investigating this question. Recent studies have identified internal directions and sparse autoencoder (SAE) features whose manipulation influences VLA behavior [4, 5, 6, 7]. These studies primarily examine nominal behavior, leaving the use of such features for adversarial defense underexplored. However, indiscriminately suppressing attackrelated features may also disrupt nominal policy behavior. An effective defense therefore requires examining both which feature to target and when to suppress it. We investigate these questions through SAE-based analysis and conditional intervention in VLA representations. We identify an SAE feature associated with patch presence and train a linear probe to detect attacks from the VLA’s internal state. At inference time, we suppress the feature’s contribution only when the probe detects an attack, while keeping the VLA parameters fixed. On LIBERO10 [8] under Untargeted Action Discrepancy Attack (UADA) [1], this conditional intervention improves task success under intermittent attacks, whereas continuous suppression of the same feature substantially degrades performance. Our results suggest that internal features associated with adversarial patches can serve as targets for VLA defense, and that intervening only during detected attacks limits disruption of nominal behavior.

2

Benign

Motivation: Attack-Related Features

Attacked

Benign

Attacked

Before introducing our method, we examine how adt=12 t=40 t=68 t=95 versarial patches are reflected in VLA representa✅ F902 Related to attacks tions. We train an SAE on internal representations collected from clean trajectories on LIBERO tasks. ✅ F1115 We then repeatedly switch an adversarial patch generated by UADA [1] on and off during a rollout in ✅ F1728 order to examine how SAE feature activations respond to patch presence. Figure 2 shows that several ❌ F1829 Unrelated to attacks SAE features track patch presence over time, a pat0 25 55 80 110 124 tern also observed across multiple episodes. These Rollout timestep t observations motivate us to investigate whether suppressing these features only when an attack is de- Figure 2: SAE features associated with adversarial patch presence. tected can improve policy robustness.

3

Methodology: Conditional Feature Intervention

Problem Setup. Our goal is to improve task success under adversarial patch attacks while limiting performance degradation under clean observations. A VLA policy generates an action at = π(ot , l) from a visual observation ot and a language instruction l. The observation contains agent-view images, wrist-view images, or both, depending on the model. We consider UADA [1], an untargeted attack that optimizes an adversarial patch δ to make predicted actions deviate from the ground= A(ot , δ), while leaving l truth actions. Applying the patch yields an attacked observation oadv t unchanged. Overview. We use a TopK SAE [9] to analyze how adversarial patches affect internal VLA representations. Our method suppresses a selected attack-related feature only when an attack is detected, aiming to improve robustness while limiting performance degradation under benign conditions. The method consists of three components: 1 identifying an attack-related feature using an SAE, 2 detecting adversarial patches with a linear probe, and 3 conditionally suppressing the selected feature based on the detection result. (i)

Attack-Related Feature Identification. Let ht,r ∈ Rd denote the representation of the rth prefix token at layer i and control step t, where r ∈ {1, . . . , Nt } and Nt is the number of prefix 1

2

(i)

tokens. We mean-pool the prefix-token representations as h̄t = N1t (i)

(i) r=1 ht,r and obtain SAE fea-

PNt

(i)

ture activations zt = E (i) (h̄t ). Using rollouts in which the adversarial patch is switched on and off, we measure the correlation between SAE feature activations and patch presence across candidate layers. We select the most consistently correlated layer-feature pair (ℓ⋆ , j ⋆ ) as the intervention (ℓ⋆ ) target and denote its decoder direction by dj ⋆ . 2 Attack Detection. Continuously suppressing the selected feature can degrade policy performance under clean observations. We therefore train a linear probe to detect patch presence from the (k) token-mean-pooled VLA representation h̄t at layer k, where k ≤ ℓ⋆ so that the detection result is (k) available before intervention. The probe estimates the attack probability as pt = σ(w⊤ h̄t + b), where σ is the sigmoid function, and w and b are learned parameters. We define the intervention gate as gt = I[pt > τ ], where I is the indicator function and τ is the detection threshold. 3 Conditional Feature Intervention. At inference time, we compute the selected feature activation from the token-mean-pooled representation and suppress its contribution in each prefix-token representation at layer ℓ⋆ : (ℓ⋆ ) (ℓ⋆ ) (ℓ⋆ ) (ℓ⋆ ) e (1) ht,r = ht,r − gt λ zt,j ⋆ dj ⋆ . |{z} | {z } detect

suppress

where λ is the intervention strength. When 0 < λ < 1, the intervention attenuates the selected feature contribution, while λ = 1 removes it completely. For λ > 1, the intervention extends beyond feature removal and steers the representation in the opposite direction along the corresponding decoder direction. The intervention is applied to all prefix tokens only when the probe detects an attack (gt = 1); otherwise, the representations remain unchanged.

4

Experiments

4.1

Experimental Setup

Models and Benchmark. We evaluate π0.5 [10] and SmolVLA [11] on LIBERO-10 [8]. For π0.5 , we use a model that uses only the agent camera, while SmolVLA uses both agent and wrist cameras. Each main evaluation setting is evaluated on 50 initial states for each of the 10 tasks, for a total of 500 rollouts. Attack Setting. We use UADA [1] targeting degree of freedom (DoF) 1 and generate one universal adversarial patch for each victim model. Our main evaluation considers intermittent attacks in which the patch appears n = 3 or n = 10 times during each rollout. Defense and Comparisons. For each victim model, we identify an attack-related SAE feature and train a logistic regression probe to detect patch presence from internal representations. At inference time, our method suppresses the selected feature only when the probe detects the patch. We compare against the attacked policy without intervention (Baseline), continuous intervention without detection (w/o Det.), and intervention applied only when the patch is actually present (w/ Oracle Det.). The probe threshold is set on separate construction data to keep the false positive rate below 2%. Evaluation. We report task success rate (SR) as the primary metric. The selected SAE feature, probe, and intervention strength are fixed before evaluation. Details of SAE training, feature selection, probe training, and intervention settings are provided in appendix B. 4.2

Main Results

Table 1: Detection performance (%).

Probe Detection Performance. We first evaluate whether the probe detects attacks during LIBERO-10 rollouts while rarely raising false positives, which would trig3

Setting

Recall

FPR

Precision

n=3 n = 10

81.76 84.91

0.39 0.71

92.71 96.45

ger unnecessary suppression. The probe uses representations from layer 5 of the π0.5 LLM backbone. As shown in Table 1, it achieves recall above 81% and precision above 92% in both settings, with a false positive rate (FPR) of 0.39% for n = 3 and 0.71% for n = 10. These FPRs remain below the 2% target used for threshold selection on separate construction data. These results support using the probe for conditional feature suppression. Effect of Conditional Intervention. Table 2 Table 2: SR (%) under attack conditions. shows that increasing patch exposure from n = Victim: π0.5 Victim: SmolVLA Method 3 to n = 10 substantially reduces baseline n = 3 n = 10 n = 3 n = 10 SR for both π0.5 and SmolVLA. Our condi- Baseline 46.8 26.0 18.6 6.2 tional intervention improves SR across all four Ours 49.2 32.4 19.6 7.6 settings, with the largest gain from 26.0% to w/o Det. 9.4 16.2 18.4 4.8 29.2 22.8 7.4 32.4% for π0.5 at n = 10. In contrast, contin- w/ Oracle Det. 52.2 uous intervention (w/o Det.) can severely degrade performance, particularly for π0.5 . Our method also achieves SR comparable to intervention based on ground-truth patch presence (w/ Oracle Det.). These results indicate that SAE feature suppression can improve robustness, but its effectiveness depends on when the intervention is applied. Performance under Benign Conditions. We eval- Table 3: SR (%) under benign conditions. uate performance under benign conditions to examine Method π0.5 SmolVLA whether the defense disrupts task execution in the abBaseline 54.4 41.4 sence of attacks. As shown in Table 3, our conditional Ours 54.6 ↑ 0.2 36.6 ↓ 4.8 intervention maintains baseline performance on π0.5 w/o Det. 35.2 ↓ 19.2 19.0 ↓ 22.4 (54.6% vs. 54.4%), while SR decreases from 41.4% to 36.6% on SmolVLA. In contrast, continuous intervention (w/o Det.) substantially reduces SR to 35.2% and 19.0%, respectively. These results indicate that restricting suppression to detected attacks is important for preventing performance degradation under benign conditions.

SR (%)

Effect of Intervention Strength. We examn=3 (oracle det.) n=10 (oracle det.) ine how the intervention strength λ affects per60 Benign (continuous) formance on π0.5 . For this analysis, separate from the main evaluation, we use 10 closed40 loop episodes per seed across five random seeds (50 rollouts total). To isolate this effect from 20 detection errors, we use oracle detection under attack and continuous intervention under be0 nign conditions. As shown in Fig. 3, under at0.1 0.2 0.3 0.5 1 2 4 Intervention Strength tack, SR peaks around λ = 0.3 for n = 3 and between λ = 0.2 and 0.25 for n = 10, then declines with stronger intervention. Under benign Figure 3: Effect of intervention strength λ on SR conditions, continuous intervention sharply re- under attack and benign conditions. duces SR as λ increases. These results reveal a trade-off between robustness under attack and benign policy performance.

5

Conclusion

We presented an SAE-based defense that suppresses an attack-related feature only when a linear probe detects an attack, without additional fine-tuning of the VLA policy. On LIBERO-10, conditional suppression improves task success under intermittent UADA attacks on π0.5 and SmolVLA, while causing less performance degradation under benign conditions than continuous suppression. Our results highlight the importance of selecting attack-related features and controlling when and how strongly to suppress them. On SmolVLA, however, the gains are modest, and benign performance still degrades. Future work should evaluate robustness to unseen attacks, and test the defense on physical robots. 4

Acknowledgments This work was supported by JST PRESTO Grant Number JPMJPR22P8, Japan.

References [1] T. Wang, C. Han, J. Liang, W. Yang, D. Liu, L. X. Zhang, Q. Wang, J. Luo, and R. Tang. Exploring the adversarial vulnerabilities of vision-language-action models in robotics. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV), pages 6948–6958. IEEE, 2025. [2] H. Xu, Y. S. Koh, S. Huang, Z. Zhou, D. Wang, J. Sakuma, and J. Zhang. Modelagnostic adversarial attack and defense for vision-language-action models. arXiv preprint arXiv:2510.13237, 2025. [3] X. Wang, M. Han, T. Hao, Y. Yang, Y.-B. Zhao, and K. Tang. Partially observable adversarial patch attacks on vision-language-action models in robotics. IEEE Robotics and Automation Letters (RA-L), 2026. [4] B. Häon, K. C. Stocking, I. Chuang, and C. Tomlin. Mechanistic interpretability for steering vision-language-action models. In Proc. Conference on Robot Learning (CoRL), 2025. [5] A. Swann, L. McGranahan, H. Buurmeijer, M. Kennedy III, and M. Schwager. Sparse autoencoders reveal interpretable and steerable features in vla models. In Proc. Conference on Robot Learning (CoRL), 2026. [6] X. Jin, A. Chatterjee, P. Kumar, and R. Paleja. Event-grounded sparse autoencoders for visionlanguage-action policies. arXiv preprint arXiv:2605.17204, 2026. [7] H. Buurmeijer, C. A. Alonso, A. Swann, and M. Pavone. Observing and controlling features in vision-language-action models. In Mechanistic Interpretability Workshop at ICML 2026, 2026. [8] B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. Proc. Annual Conference on Neural Information Processing Systems (NeurIPS), 36:44776–44791, 2023. [9] L. Gao, T. Dupre la Tour, H. Tillman, G. Goh, R. Troll, A. Radford, I. Sutskever, J. Leike, and J. Wu. Scaling and evaluating sparse autoencoders. In Proc. International Conference on Learning Representations (ICLR), volume 2025, pages 26721–26754, 2025. [10] K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. R. Equi, C. Finn, N. Fusai, M. Y. Galliker, D. Ghosh, L. Groom, K. Hausman, b. ichter, S. Jakubczak, T. Jones, L. Ke, D. LeBlanc, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, A. Z. Ren, L. X. Shi, L. Smith, J. T. Springenberg, K. Stachowicz, J. Tanner, Q. Vuong, H. Walke, A. Walling, H. Wang, L. Yu, and U. Zhilinsky. π0.5 : a vision-language-action model with open-world generalization. In Proc. Conference on Robot Learning (CoRL), pages 17–40. PMLR, 2025. [11] M. Shukor, D. Aubakirova, F. Capuano, P. Kooijmans, S. Palma, A. Zouitine, M. Aractingi, C. Pascal, M. Russi, A. Marafioti, et al. Smolvla: A vision-language-action model for affordable and efficient robotics. arXiv preprint arXiv:2506.01844, 2025. [12] T. B. Brown, D. Mané, A. Roy, M. Abadi, and J. Gilmer. Adversarial patch. arXiv preprint arXiv:1712.09665, 2017. [13] X. Liu, H. Yang, Z. Liu, L. Song, H. Li, and Y. Chen. Dpatch: An adversarial patch attack on object detectors. arXiv preprint arXiv:1806.02299, 2018. 5

[14] K. Eykholt, I. Evtimov, E. Fernandes, B. Li, A. Rahmati, C. Xiao, A. Prakash, T. Kohno, and D. Song. Robust physical-world attacks on deep learning visual classification. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1625– 1634. IEEE, 2018. [15] A. Athalye, L. Engstrom, A. Ilyas, and K. Kwok. Synthesizing robust adversarial examples. In Proc. International Conference on Machine Learning (ICML), pages 284–293. PMLR, 2018. [16] C. Xiang, A. N. Bhagoji, V. Sehwag, and P. Mittal. Patchguard: A provably robust defense against adversarial patches via small receptive fields and masking. In 30th USENIX Security Symposium (USENIX Security 21), pages 2237–2254, 2021. [17] C. Xiang, S. Mahloujifar, and P. Mittal. Patchcleanser: Certifiably robust defense against adversarial patches for any image classifier. In 31st USENIX security symposium (USENIX Security 22), pages 2065–2082, 2022. [18] L. Jing, R. Wang, W. Ren, X. Dong, and C. Zou. Pad: Patch-agnostic defense against adversarial patch attacks. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 24472–24481. IEEE, 2024. [19] Y. Yan, Y. Xie, Y. Zhang, L. Lyu, H. Wang, and Y. Jin. When alignment fails: Multimodal adversarial attacks on vision-language-action models. arXiv preprint arXiv:2511.16203, 2025. [20] H. Lu, Y. Yu, Y. Yang, C. Yi, Q. Zhang, B. Shen, A. C. Kot, and X. Jiang. When robots obey the patch: Universal transferable patch attacks on vision-language-action models. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22867– 22878, 2026. [21] R. Huben, H. Cunningham, L. Smith, A. Ewart, and L. Sharkey. Sparse autoencoders find highly interpretable features in language models. In Proc. International Conference on Learning Representations (ICLR), volume 2024, pages 7827–7845, 2024. [22] H. Lim, J. Choi, J. Choo, and S. Schneider. Sparse autoencoders reveal selective remapping of visual concepts during adaptation. In Proc. International Conference on Learning Representations (ICLR), volume 2025, pages 24444–24469, 2025. [23] V. Zaigrajew, H. Baniecki, and P. Biecek. Interpreting CLIP with hierarchical sparse autoencoders. In Proc. International Conference on Machine Learning (ICML), 2025. [24] H. Lou, C. Li, J. Ji, and Y. Yang. SAE-v: Interpreting multimodal models for enhanced alignment. In Proc. International Conference on Machine Learning (ICML), 2025. [25] L. Hufe, C. Venhoff, E. Purelku, M. Dreyer, S. Lapuschkin, and W. Samek. Dyslexify: A mechanistic defense against typographic attacks in CLIP. In Proc. International Conference on Learning Representations (ICLR), 2026. [26] S. Joseph, P. Suresh, E. Goldfarb, L. Hufe, Y. Gandelsman, R. Graham, D. Bzdok, W. Samek, and B. A. Richards. Steering clip’s vision transformer with sparse autoencoders. arXiv preprint arXiv:2504.08729, 2025. [27] K. O’Brien, D. Majercak, X. Fernandes, R. G. Edgar, B. Bullwinkel, J. Chen, H. Nori, D. Carignan, E. Horvitz, and F. Poursabzi-Sangdeh. Steering language model refusal with sparse autoencoders. In ICML 2025 Workshop on Reliable and Responsible Foundation Models, 2025. [28] H. Wang, G. Wang, and H. Zhang. Steering away from harm: An adaptive approach to defending vision language model against jailbreaks. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 29947–29957, 2025. doi: 10.1109/CVPR52734.2025.02787. 6

[29] B. W. Lee, I. Padhi, K. Natesan Ramamurthy, E. Miehling, P. Dognin, M. Nagireddy, and A. Dhurandhar. Programming refusal with conditional activation steering. In Proc. International Conference on Learning Representations (ICLR), volume 2025, pages 90960–90985, 2025.

7

A

Related Work

A.1

Adversarial Patch Attacks and Defenses.

Adversarial patches are localized visual patterns that alter the predictions of image classifiers and object detectors [12, 13]. Prior work has made such perturbations robust to changes in viewing conditions, including physically realizable ones [14, 15]. Defenses include masking corrupted features [16], certified input masking [17], and patch localization and removal [18]. In VLA models, visual perturbations can affect successive actions and disrupt task execution. Wang et al. [1] introduced patch attacks that either increase action discrepancies, as in UADA, or manipulate robot trajectories. Subsequent studies attack visual embeddings and image–instruction alignment [2], design attacks spanning textual, visual, and cross-modal perturbations [19], craft universal patches that transfer across models, tasks, and viewpoints [20], and optimize patches using only a short prefix of an execution trajectory [3]. Existing VLA defenses either preprocess the input images or fine-tune the model. JPEG compression and Gaussian blur provide limited protection against the patch attacks studied in [3]. Xu et al. [2] adversarially fine-tune the visual encoder to align the representations of clean and perturbed inputs. Our method instead intervenes directly in internal VLA representations at inference time, keeping the input images and policy parameters unchanged. A.2

Mechanistic Interpretability and Representation Intervention.

Sparse autoencoders (SAEs) decompose model activations into sparse features that can support mechanistic analysis [21, 9]. Their applications have extended from language models to visual concept analysis in CLIP [22, 23] and to cross-modal representation analysis in multimodal models [24]. Recent work has brought mechanistic analysis to VLA policies. Directions associated with motion properties can steer robot behavior at inference time [4]. SAE-based studies identify features associated with motion primitives, semantic concepts, and behavioral events, and test their influence through closed-loop interventions [5, 6]. These studies primarily examine nominal behavior, leaving the use of such features for adversarial defense underexplored. Internal interventions have also served as defenses in vision and language models. In CLIP, Dyslexify [25] improves robustness to typographic attacks without fine-tuning by ablating attention heads that transmit typographic information to the CLS token. Suppressing specific SAE features in CLIP’s vision transformer also mitigates typographic attacks [26]. Such interventions can also affect benign inputs. In language models, amplifying refusal-related SAE features can improve resistance to jailbreaks but can also degrade performance on benign inputs [27]. Some studies limit this degradation. Adaptive activation steering defends vision-language models against jailbreaks from adversarial images by removing activation directions associated with harmful responses, adjusting the amount of removal for each input [28]. Conditional activation steering uses properties detected in hidden states to decide when to steer [29], and O’Brien et al. [27] show that conditioning SAE steering on a prompt-safety classifier mitigates the regressions on benign inputs. Our work brings this conditional approach to adversarial patch defense in VLA models. We suppress an SAE feature associated with patch presence only when a linear probe on the VLA’s hidden state detects an attack. We evaluate this intervention in closed-loop robot control and measure its effect on task success under both adversarial and benign conditions.

B

Experimental Details

Models. We evaluate two VLA models, π0.5 [10] and SmolVLA [11], on the LIBERO-10 benchmark [8]. For π0.5 , we fine-tune the π0.5 -base checkpoint on the LIBERO training split following the official low-rank adaptation (LoRA) fine-tuning procedure. To obtain a policy that uses only 8

the agent camera, we replace all images from the wrist camera with black images during finetuning. Fine-tuning runs for 30,000 steps on four NVIDIA RTX 6000 Ada GPUs (48 GB each). For SmolVLA, we use the checkpoint released on Hugging Face without further fine-tuning. The weights of both policies remain fixed in all defense experiments. Patches. Fig. 4 shows the patches used in our experiments. Our main attack is UADA [1]. For each victim model, we optimize one universal patch on LIBERO-10 demonstration trajectories, following the original UADA implemenπ0.5 SmolVLA tation and optimization settings, including the (a) UADA (b) Gripper patch size and geometric transformations. The Figure 4: Adversarial patches. attack targets DoF 1 of the robot action. Each optimized patch is shared across all LIBERO-10 tasks and evaluation rollouts and is applied to every camera view that the victim model uses. For the additional evaluation in Appendix C, we also use a gripper patch, inspired by VLA-Fool [19]: an image of a robot gripper that we insert into the observation without optimizing it against the victim model. In intermittent attacks, a patch is visible during n ∈ {3, 10} intervals of 10 consecutive frames each and absent otherwise. SAE Training. Following the Dr. VLA implementation [5], we train a separate TopK SAE [9] for each candidate layer. The training data are representations from the residual stream, collected during clean LIBERO rollouts. At each sampled control step, we mean-pool the representation over tokens and use the resulting vector as one training example. As in Dr. VLA, we use a 1× expansion ratio, so the SAE latent dimension equals the dimension of the VLA representation. We use a smaller TopK sparsity than Dr. VLA, keeping K = 12 active latents for π0.5 and K = 6 for SmolVLA. We train each SAE for 6,677 optimization steps and follow Dr. VLA for all other architecture and optimization settings. Attack-Related Feature Selection. We select the intervention target (ℓ⋆ , j ⋆ ) on 10 development rollouts, one per LIBERO-10 task, that are separate from the main evaluation. In each rollout, the patch is inserted at randomly sampled intervals. For each candidate layer-feature pair, we compute the Pearson correlation between the feature activation and the binary patch-presence indicator, separately for each rollout. We then select the pair whose correlation exceeds 0.6 in the largest number of rollouts and break ties by the mean correlation across rollouts. Linear Probe Training. We train and evaluate probes on 100 rollout pairs, each consisting of a clean and an attacked rollout from the same initial state. These rollouts are separate from those used for feature selection and from the 500 evaluation rollouts, and we split them 60/20/20 into training, development, and test sets within each task. Frames without a visible patch are labeled negative and frames with a visible patch are labeled positive; each attacked rollout contains three 10-frame patch intervals. For π0.5 , the probe is a class-balanced logistic regression on the token-mean-pooled prefix representation at layer 5 of the PaliGemma backbone (2048 dimensions), with the inverse regularization strength C chosen from {0.01, 0.1, 1, 10, 100}. For SmolVLA, the probe is a linear classifier trained with class-balanced binary cross-entropy and L2 regularization on the mean over prefix tokens, excluding padding, at layer 9 of the VLM backbone. For both models, we set the threshold τ on clean development data so that the FPR is at most 2%. Under this constraint, we select SmolVLA’s C to maximize the detection rate on patch intervals of 10 frames in the development set. We also train probes independently at other layers of the VLM and the vision encoder. Intervention Strength. We apply Eq. (1) at the selected pair (ℓ⋆ , j ⋆ ) and set the strength λ separately for each experimental condition. We search λ ∈ {0.25, 0.5, 1, 2, 4}, adding {0.1, 0.2, 0.3} in finer-grained experiments, select λ on the development rollouts, and fix it before evaluation. In the main evaluation, we use λ = 0.3 for n = 3 and λ = 0.2 for n = 10 on π0.5 , and λ = 0.2 for n = 3 and λ = 0.25 for n = 10 on SmolVLA. For SmolVLA, we use λ = 0.25 under benign conditions 9

and λ = 0.1 under persistent attacks. Fig. 3 reports the sensitivity of SR to λ on π0.5 under both attack and benign conditions.

C

Additional Experiments

We further analyze our method beyond the main evaluation. We evaluate (1) performance under gripper patch attacks, (2) detector specificity across patch types, (3) the effect of attack-related feature selection, and (4) performance under persistent attacks. C.1

Evaluation under Gripper Patch Attacks

Table 4: SR (%) under gripper patch attacks.

We evaluate π0.5 under gripper patch attacks, in which the Victim: π0.5 Method inserted patch is an unoptimized image of a robot gripper n = 3 n = 10 (Fig. 4b). For this setting, we train the probe on rollouts Baseline 30.4 4.4 with the gripper patch. The unoptimized gripper patch Ours 46.4 33.6 lowers baseline SR more than UADA does: to 30.4% for w/o Det. 0.0 0.0 w/ Oracle Det. 50.0 31.8 n = 3 and 4.4% for n = 10, compared with 46.8% and 26.0% under UADA (Tables 4 and 2). Our method raises SR to 46.4% and 33.6%, respectively, which is comparable to oracle detection (50.0% and 31.8%). Intervention without detection reduces SR to 0% in both settings. As with UADA, controlling when to intervene remains important under this patch type. C.2

Detector Specificity Across Patch Types

Table 5: Detection performance (%) across patch types.

To measure how specific the probe is to its train- Training Patch Test Patch Recall FPR Precision ing patch type, we evaluate it on white, black, UADA 81.76 0.39 92.71 and random-noise patches (Table 5). The probe White 5.74 1.47 79.59 UADA Black 2.17 1.47 59.56 trained on UADA patches rarely flags white or Noise 100.00 1.47 98.55 black patches (recall 5.74% and 2.17%) but flags Gripper 84.91 0.71 96.45 every frame with a random-noise patch (recall White 23.11 9.45 70.97 Gripper 100%). The UADA probe therefore detects ranBlack 16.32 9.45 63.33 Noise 19.71 9.45 67.59 dom noise as well as UADA patches. The probe trained on gripper patches detects 16.32–23.11% of white, black, and noise patches, with a higher FPR of 9.45%. These differences suggest that detector specificity depends on the patches used to train the probe. Effect of Attack-Related Feature Selection

We test how the choice of the suppressed feature affects SR. For π0.5 at n = 10, we compare the selected feature with SAE features sampled randomly from layers deeper than the detection layer, keeping the detection and intervention settings identical. For each feature, we measure its mean Pearson correlation with patch presence and the SR obtained by suppressing it. As shown in Fig. 5, the selected feature has the highest correlation and yields the highest SR. Some random features also improve SR over the baseline, but by smaller margins, and others reduce it. These results support selecting the suppressed feature by its correlation with patch presence.

10

34 32 30

SR (%)

C.3

28 26 24

Random features Selected feature Baseline

22 20

0.2

0.1

0.0

0.1

0.2

0.3

Mean Pearson correlation

0.4

0.5

Figure 5: Correlation with patch presence and SR of the selected and random features (π0.5 , n = 10).

C.4

Performance under Persistent Attacks

We also evaluate a persistent attack in which the patch re- Table 6: SR (%) under persistent atmains visible throughout the rollout. Because the patch tacks. Method π0.5 SmolVLA appears in every frame, oracle detection is equivalent to Baseline 12.4 1.8 continuous intervention in this setting. As shown in TaOurs 9.0 ↓ 3.4 1.6 ↓ 0.2 ble 6, no intervention method improves SR. For π0.5 , our w/o Det. 7.4 ↓ 5.0 1.6 ↓ 0.2 method lowers SR from 12.4% to 9.0%, and both w/o w/ Oracle Det. 7.4 ↓ 5.0 1.6 ↓ 0.2 Det. and w/ Oracle Det. lower it to 7.4%. For SmolVLA, all three intervention methods yield 1.6%, compared with 1.8% for the baseline. Even with oracle detection, suppressing the selected feature does not recover SR under persistent attacks, which is a limitation of our approach.

11

Related documents

• Detect and Suppress: A Mechanistic Defense against Adversarial Patches in VLA Models#1188176
• PHANTOM: A Large-Scale Dataset of Multimodal Adversarial Attacks for Vision-Language Models#303213
• Visual Adversarial Attack on Vision-Language Models for Autonomous Driving#155515
• AEGIS: Anchor-Enforced Gradient Isolation for Knowledge-Preserving Vision-Language-Action Fine-Tuning#1156330
• Hearing the Unspoken: Language Model Priors for Acoustic Adversarial Attacks#266128