arXiv CSopen access

AirflowAttack: Thermal-Airflow Adversarial Perturbations against Infrared Remote-Sensing Vision-Language Models

AirflowAttack: Thermal-Airflow Adversarial Perturbations against Infrared Remote-Sensing Vision-Language Models Cong Su2 , Jiaju Han1 , Xuemeng Sun1 , Chengyin Hu1 , Qike Zhang1 , Jiujiang Guo2 , Yiwei Wei1 , and Jiahuan Long3

arXiv:2607.06485v1 [cs.CV] 7 Jul 2026

1

China University of Petroleum-Beijing at Karamay, Karamay, Xinjiang, China 2 Tianjin University, Tianjin, China 3 Shanghai Jiao Tong University, Shanghai, China

Abstract. Vision-language models (VLMs) are increasingly deployed on infrared (IR) remote sensing imagery in security-critical settings, yet their adversarial robustness remains unexamined. We present AirflowAttack, to our knowledge the first adversarial attack for IR remotesensing VLMs and the first to weaponize thermal-airflow turbulence as the perturbation prior. A lightweight generator synthesizes a single inputagnostic perturbation regularized toward physically plausible airflow patterns. Optimized on one surrogate CLIP model, it attains a mean zeroshot scene-classification attack success rate (ASR, the fraction of samples whose top-1 class changes) of 48.5% across five diverse CLIP backbones, far exceeding four IR-specific physical baselines (27.7–37.0%). Applied to six state-of-the-art VLMs, it cuts scene-classification accuracy by up to 38.2% (relative)—yet paradoxically makes some models more confident in their IR analysis, confabulating the perturbation as genuine thermal evidence such as temperature gradients and convection. Ablations show the airflow prior raises physical plausibility at no measurable cost to attack success. Together with a benchmark spanning eleven models and four tasks, these findings expose critical vulnerabilities in the rapidly expanding IR VLM ecosystem. Keywords: Adversarial attack · Infrared remote sensing · Vision-language model · Transferable perturbation · Thermal airflow

1

Introduction

Infrared remote sensing underpins critical applications from disaster monitoring and environmental surveillance to military reconnaissance, operating under conditions where visible-spectrum imaging fails—nighttime, fog, smoke, and thermal camouflage detection. The recent adaptation of vision-language models (VLMs) to the IR domain promises a step change in automated scene understanding: models such as GeoRSCLIP [46], RemoteCLIP [21], and RS5M [47]

2

C. Su et al. Parameters

Random

Initial

1. Classification

Final Perturbation

AdamW Optimizer

Lowdimensional residual 𝑍

Adversarial Image

𝐦𝐢𝐧 𝓛 𝒁, 𝒂 = 𝟖𝓛𝓬𝓸𝓷𝓯 𝒙𝒂𝒅𝒗 + 𝓛𝓪𝓲𝓻 𝜹, 𝑷𝒂𝒊𝒓 𝒁,𝒂

Adv image

…

Global amplitude 𝑎

…

…

…

Lr:0.095 Step:800

Z

Momentum update

Top-1 Prediction

…

…

Inconformity

a

Ground Truth

2. Caption

Airflow Perturbation Synthesis Module:

𝒙𝒂𝒅𝒗 = clip 𝒙 + 𝑮 ⊙ 𝝐 ⋅ 𝒕𝒂𝒏𝒉 𝒂𝑷𝒂𝒊𝒓 + 𝟎. 𝟔 ⋅ Blur Upsample 𝒁

(1) Airflow Magnitude Mapping for Spatial Gating Positive Thermal Perturbation

Negative Thermal Perturbation

Adv image

Clean

Learnable residual grid

Magnitude Extraction and Gate Formation

Each cell controls a local airflow adjustment

Positive Thermal Perturbation

𝑮 = 𝑺𝒎𝒐𝒐𝒕𝒉 𝑷air

Signed Airflow Prior

, 𝟎, 𝟏

(2) Low-Dimensional Residual for Coarse Airflow Control

Residential are a with tennis court, buildings, roads, and trees.

Negative Thermal Perturbation

ChatGPT Scoring

𝑍 ∈ 𝑅𝟛𝟚×𝟛𝟚 Coarse-grained control

Spatial Airflow Gate

(3) Upsampling and Smoothing to Continuous Residual Field

Intersec tion with vehic le s, buildings, roads, and bright infrared spots..

3. VQA What is the main scene type in this infrared remote sens ing image?

𝑹 = Blur Upsample 𝒁

Residentia l area

Low−dimensional residual Z

Upsample

Gaussian smoothing

Continuous residual field R

Positive

…… What is the main scene type in this infrared remote sens ing image? VQAResidential area with roads, vehicles, and s tructural outlines .

ChatGPT Scoring

Coarse local residual

Expand to image resolution

Remove block artifacts

Smooth local airflow refinements

Negative

Fig. 1: Overview of AirflowAttack. A lightweight generator Gθ maps a low-dimensional latent code to a single-channel thermal-airflow perturbation, optimized on a surrogate IR-finetuned CLIP model under an L∞ ≤ ε constraint using a confidence loss Lconf and an airflow-correlation loss Lair . The resulting perturbation transfers, without targetmodel access, to five CLIP backbones and six VLMs across four vision-language tasks.

can now jointly reason about IR imagery and natural language, enabling openvocabulary retrieval, descriptive captioning, and visual question answering over thermal scenes. However, the security implications of deploying VLMs in IRsensitive contexts remain entirely unexamined. Adversarial attacks—imperceptible input perturbations that cause models to fail—have been extensively studied in the RGB domain, spanning white-box [15], black-box [29], and universal [27] regimes. Yet IR imagery differs fundamentally from RGB: thermal sensors capture emitted radiation rather than reflected light, producing single-channel intensity maps governed by Planck’s law where pixel values encode physical temperature. This physical grounding both constrains and motivates a new class of attacks: rather than crafting arbitrary digital noise, an adversary can simulate physically plausible thermal phenomena—such as airflowinduced temperature distortions—that are simultaneously harder to detect and more likely to transfer across models. In this paper, we introduce AirflowAttack, to the best of our knowledge the first adversarial attack designed for IR remote-sensing VLMs. While universal perturbations have been studied for unimodal remote-sensing classifiers [41] and for RGB vision-language models [45, 49], and while simulated atmospheric phenomena such as haze [14] and weather [32] have been repurposed as attacks in the RGB domain, none of these targets the thermal-IR modality, the remotesensing VLM setting, or thermal-airflow turbulence as the perturbation prior— the specific intersection our work addresses. Our approach synthesizes an input-

Thermal-Airflow Attack against IR Remote Sensing VLMs

3

agnostic adversarial perturbation by modeling temperature fluctuations induced by thermal airflow turbulence. Unlike conventional pixel-space UAPs that produce unstructured noise, the proposed perturbation mimics the spatially correlated, physically grounded patterns of atmospheric thermal mixing—making it simultaneously effective, transferable across architectures, and physically interpretable as a natural thermal phenomenon (see Fig. 1). We conduct a comprehensive adversarial robustness evaluation spanning five CLIP-family backbones (OpenAI-CLIP-B32, OpenAI-CLIP-L14, OpenCLIP-B32, RemoteCLIP-B32, GeoRSCLIP-B32) and six state-of-the-art VLMs (Qwen2.5VL-7B, InstructBLIP, LLaVA-1.5, LLaVA-1.6, GeoChat, H2RSVLM) across four vision-language tasks on a dedicated 1,000-sample infrared test set. Our contributions are: – We propose AirflowAttack, to our knowledge the first adversarial attack for IR remote-sensing VLMs, synthesizing a transferable thermal-airflow perturbation via a lightweight generator. – Across five CLIP backbones and six VLMs, a single surrogate-optimized perturbation attains 48.5% mean ASR, exceeding four IR-specific baselines and transferring without target access. – Extensive ablations reveal the mechanisms of airflow-based transfer, including a paradoxical rise in some models’ IR-cue confidence under attack.

2

Related Work

2.1

Infrared Remote Sensing Vision-Language Models

CLIP [30] showed that contrastive image-text pretraining yields strong zeroshot transfer, a paradigm extended to remote sensing by RemoteCLIP [21], GeoRSCLIP [46], and RS5M [47], and by general-purpose VLMs such as LLaVA1.5/1.6 [22,23], InstructBLIP [12], Qwen2.5-VL [2], GeoChat [19], and H2RSVLM [28] applied to RS scene classification, detection, VQA, and captioning. These models are developed and evaluated under benign conditions. While the robustness of unimodal RS classifiers has been studied [40, 41], and very recent work crafts transferable adversarial examples for RS object recognition [20] and attacks infrared VLMs with physical patches [16], the robustness of IR remote-sensing VLMs to input-agnostic perturbations remains unexamined. 2.2

Adversarial Attacks

Since neural networks were shown vulnerable to imperceptible perturbations [33], white-box attacks (FGSM [15], C&W [7], PGD [25], AutoAttack [10]) and transferboosting methods (MI-FGSM [13], DIM [38]) have been widely studied. Universal perturbations [27] fool a model across inputs with a single pattern, and have been extended to remote sensing [41] and to CLIP/VLP models (AdvCLIP [49], Co-Attack [45], and recent targeted VLM attacks [6]). These target unimodal RGB classifiers or RGB CLIP; physically realizable RGB attacks [1, 5] likewise

4

C. Su et al.

exploit color/texture cues absent in thermal imagery. Our perturbation instead targets the single-channel thermal modality and is parameterized to resemble a physical thermal phenomenon. 2.3

Physical Attacks on Thermal Imaging

A parallel line of work explores physical attacks specific to thermal sensors. Methods include projecting thermal patterns using controlled heat sources [37], physically adversarial infrared patches with learnable shapes and locations [51], adversarial infrared curves and grid patterns against pedestrian detectors [17,34], wearable hot/cold blocks that fool thermal detectors [36], exploiting sensor nonuniformity [4], and introducing fixed-pattern noise [3]. A recent survey [11] provides a broader taxonomy of IR-specific adversarial methods. A closely related line simulates a physical atmospheric phenomenon and repurposes it as an adversarial perturbation—adversarial haze [14], adversarial weather [32], and, concurrently, physically-induced atmospheric perturbations for RS classification [52]; our airflow-turbulence perturbation is a new instance of this template, uniquely targeting the thermal-IR VLM setting. Atmospheric turbulence, caused by spatial and temporal variations in air refractive index due to temperature gradients, is a well-known degradation in long-range thermal imaging [50]; learned turbulence simulation models [26] have improved the fidelity of synthetic turbulence. However, prior work treats turbulence as a nuisance to be corrected, not as an adversarial primitive to be exploited. Our work is the first to synthesize thermal airflow turbulence patterns as a deliberate universal perturbation optimized for attacking multimodal IR models. 2.4

VLM Robustness

Recent studies have begun examining the adversarial robustness of VLMs [31,48]. Adversarial images can cause VLMs to produce hallucinated captions [44], and visual adversarial perturbations can transfer to the language modality [39]. Setof-mark prompting [42] further reveals how VLMs attend to image regions, which is directly relevant to understanding why spatially localized perturbations matter. However, existing work focuses almost exclusively on RGB inputs and generic multimodal reasoning. The IR modality introduces distinct challenges: singlechannel intensity encoding, different feature statistics, and physical constraints on plausible perturbations. To the best of our knowledge, this work is the first to study adversarial robustness for IR remote-sensing VLMs.

3

Method

We propose AirflowAttack, a framework for generating universal adversarial perturbations that simulate thermal airflow turbulence to attack IR vision-language models. The key insight is that physically interpretable thermal patterns—unlike arbitrary pixel noise—exploit domain-specific feature representations and transfer more effectively across architectures.

Thermal-Airflow Attack against IR Remote Sensing VLMs

3.1

5

Threat Model

We consider a gray-box adversary with access to a surrogate CLIP model (OpenAICLIP-B32, IR-finetuned) but no access to target models, their parameters, or training data. The adversary can perturb input IR images before they are processed by downstream VLMs. The perturbation must satisfy an L∞ constraint: ∥δ∥∞ ≤ ε, where ε = 100 in pixel intensity space (out of 255). The adversary’s goal is to craft a single universal perturbation δ that, when added to any IR image x, causes incorrect scene classification, captioning, or VQA outputs across multiple target models. This is a realistic threat: IR sensors deployed in the field may process adversarially perturbed inputs before human or automated analysis. 3.2

Thermal Airflow Perturbation Model

Rather than optimizing δ directly in pixel space, we parameterize the perturbation through a lightweight generative model Gθ (z) that maps a low-dimensional latent vector z ∈ Rd to a full-resolution perturbation pattern. This parameterization serves three purposes: it regularizes the perturbation toward spatially coherent patterns, it reduces the optimization search space, and it enables the integration of physical priors through architectural design. The generator consists of a series of transposed convolution layers with residual connections, mapping a latent code of dimension d = 32 to a single-channel output matching the input resolution. We employ a residual ratio of r = 0.60, meaning that 60% of the perturbation energy passes through the residual pathway, which we find empirically balances attack strength against physical plausibility (see Sec. 4.5). For each training iteration, a latent vector z is sampled and decoded to produce the perturbation δ = Gθ (z). The perturbation is then scaled to satisfy the L∞ constraint: δ ← clip(δ, −ε, ε). The adversarial image is formed as xadv = x + δ. 3.3

Optimization Objective

The optimization objective combines an attack loss that drives retrieval failure with a correlation loss that enforces physical plausibility. Given an IR image x, its paired caption t+ , and a set of distractor captions T − , we define the confidence (contrastive image–text alignment) loss: \mathcal {L}_{\text {conf}} = -\log \frac {\exp (\tau \cdot \text {sim}(f_I(x^{\text {adv}}), f_T(t^+)))}{\sum _{t \in \{t^+\} \cup \mathcal {T}^-} \exp (\tau \cdot \text {sim}(f_I(x^{\text {adv}}), f_T(t)))} \label {eq:conf_loss}

(1)

where fI and fT are the image and text encoders of the surrogate CLIP model, sim(·, ·) denotes cosine similarity, and τ is a temperature parameter. Minimizing Lconf reduces the similarity between the adversarial image embedding and its correct caption while increasing similarity with distractors. To encourage the perturbation to resemble natural thermal airflow, we introduce an airflow correlation loss:

6

C. Su et al.

Algorithm 1: AirflowAttack universal perturbation optimization Input: Surrogate CLIP encoders fI , fT ; IR pairs {(x, t+ )} with distractors T − ; airflow templates A; budget ε, steps T , rate η, weights α, β Output: Universal perturbation δ ⋆ 1 θ, z ← Init // generator params and latent code 2 for i = 1 . . . T do
3 δ ← clip Gθ (z), −ε, ε // decode, project to L∞ ball 4 xadv ← x + δ exp(τ sim(fI (xadv ), fT (t+ ))) // attack loss 5 Lconf ← − log P adv ), f (t))) T t∈{t+ }∪T − exp(τ sim(fI (x 6 Lair ← 1 − Corr(δ, A) // airflow-plausibility loss 7 L ← α Lconf + β Lair 8 (θ, z) ← (θ, z) − η ∇(θ,z) L // gradient step 9 end
⋆ 10 δ ← clip Gθ (z), −ε, ε ⋆ 11 return δ

\mathcal {L}_{\text {air}} = 1 - \text {Corr}(\delta , \mathcal {A}) \label {eq:air_loss}

(2)

where A is a bank of synthetic thermal airflow templates generated by simulating atmospheric turbulence via randomized heat kernel convolution, and Corr(·, ·) denotes spatial Pearson correlation. This term penalizes perturbations that deviate from physically plausible airflow patterns. The total loss is a weighted combination: \mathcal {L} = \alpha \cdot \mathcal {L}_{\text {conf}} + \beta \cdot \mathcal {L}_{\text {air}} \label {eq:total_loss}

(3)

with α = 8 and β = 2, determined through ablation (Sec. 4.5). Optimization proceeds via projected gradient descent over the generator parameters θ and latent code z for 800 steps with learning rate η = 0.055. Algorithm 1 summarizes the full procedure.

3.4

Surrogate-to-Target Transfer

Once optimized on the surrogate model, the final UAP δ ∗ = Gθ∗ (z ∗ ) is applied directly to any target IR image. No access to target models is required at deployment time. The perturbation transfers effectively because: (i) the airflow pattern encodes domain-general thermal features that multiple IR-trained models have learned to associate with atmospheric conditions; (ii) the universal nature of the perturbation means it does not overfit to surrogate-specific decision boundaries; and (iii) the L∞ constraint ensures the perturbation magnitude is consistent across inputs, avoiding the brittleness of per-image attacks.

Thermal-Airflow Attack against IR Remote Sensing VLMs

4

Experiments

4.1

Experimental Setup

7

Models. We evaluate five CLIP-family backbones spanning different architectures, pretraining data, and IR adaptation strategies: OpenAI-CLIP-B32, OpenAICLIP-L14 [30], OpenCLIP-B32 [9], RemoteCLIP-B32 [21], and GeoRSCLIPB32 [46]. For VLM evaluation, we use six state-of-the-art models: Qwen2.5-VL7B [2], InstructBLIP [12], LLaVA-1.5 (7B) [23], LLaVA-1.6 (7B) [22], GeoChat (7B) [19], and H2RSVLM [28]. Datasets. Our dataset is constructed from five public RS sources (NWPUCaption [8], RSICD [24], RSITMD [43], RS5M [47], SkyScript [35]), filtered to infrared samples to prevent RGB information leakage. The filtering procedure removes samples whose image filenames or metadata contain RGB indicators. Final class distribution and per-source proportions are detailed in the supplementary material. The test split contains 10,000 IR image-text pairs, of which 9,720 carry a remote-sensing scene-category label and are used for the CLIP zeroshot classification attack; the validation split (used for ablations) contains 416 samples. The VLM evaluation uses 1,000 randomly sampled labeled IR images with annotations for four tasks. Metrics. For CLIP, each backbone is a zero-shot scene classifier over remotesensing scene categories: each IR image is assigned the scene class whose text prompt has the highest image–text cosine similarity. We report Attack Success Rate (ASR): the percentage of samples whose adversarial top-1 scene prediction differs from the clean top-1 prediction.4 For the cross-model transfer study (Sec. 4.2), where a single perturbation is applied to held-out samples and evaluated by nearest-caption retrieval over a 1,000-sample candidate pool, we additionally report retrieval top-1 flip rate, confidence drop, and the number of unique adversarial top-1 captions as an error-diversity measure. For VLM evaluation, we report ROUGE-L for caption quality, scene accuracy, object F1, and IR-cue accuracy. All metrics use the full evaluation set. For VLM experiments, we assess every clean-vs-attack change with a two-proportion test (n=1000) and Wilson 95% confidence intervals, correcting 18 comparisons with both Bonferroni and Benjamini–Hochberg (FDR) procedures. Because clean and adversarial predictions are evaluated on the same images, this unpaired test treats them as independent and is therefore conservative relative to an exact paired (McNemar) test; the significance markers we report thus understate rather than overstate the effects. Baselines. We compare against four IR-specific physical attack methods: (1) Atmospheric thermal turbulence—adding synthetic turbulence patterns generated by phase-screen propagation models [26] (distinct from our airflow template bank, which uses randomized heat-kernel convolution, so that the baseline is an independent turbulence formulation rather than a weakened version of our 4

ASR is a paired flip rate—clean vs. adversarial prediction on the same image—so the clean condition is the reference (0% by construction) and no separate clean-accuracy baseline is required. The denominator is the full evaluated set.

8

C. Su et al.

Table 1: CLIP zero-shot scene-classification attack success rate (ASR %) across five backbones at ε = 100. ASR is the fraction of test samples whose top-1 predicted scene class under attack differs from the clean prediction. AirflowAttack exceeds all four IR-specific physical baselines on every backbone. Method

OpenAI-CLIP-L14 OpenAI-CLIP-B32 OpenCLIP-B32 RemoteCLIP-B32 GeoRSCLIP-B32

Atmospheric turbulence Stripe noise Hot/cold blocks Sensor drift

36.9 28.2 25.2 24.9

35.6 39.5 29.6 32.1

36.1 38.3 28.5 25.9

28.8 41.3 31.8 27.9

30.8 37.8 30.7 27.7

AirflowAttack (Ours)

38.3

54.2

47.9

50.6

51.7

prior); (2) IR fixed-pattern stripe noise—adding periodic column-wise noise mimicking sensor readout artifacts; (3) Thermal hot/cold blocks—overlaying rectangular regions with elevated or depressed temperatures, following the HOTCOLD block design [36]; (4) Thermal sensor drift/non-uniformity—applying pixel-wise gain and offset variations simulating detector degradation. All baselines operate under the same ε = 100 constraint. We deliberately restrict the comparison to IR-specific physical attacks, which share AirflowAttack’s physical-plausibility motivation; unstructured digital baselines (pixel-space UAP, Gaussian noise) address the orthogonal question of whether perturbation structure matters, which we examine directly through the loss-component and spatial-position ablations in Sec. 4.5.

4.2

Experiment 1: Zero-Shot Scene Classification Attack

Figure 2 shows representative zero-shot classification flips, and Tab. 1 reports the ASR of AirflowAttack compared to four IR-specific baselines across five CLIP backbones. AirflowAttack achieves the highest ASR on every backbone, with a mean ASR of 48.5% versus 33.6% (turbulence), 37.0% (stripe noise), 29.2% (hot/cold blocks), and 27.7% (sensor drift). The strongest result is on OpenAICLIP-B32 (54.2%), the surrogate model, but high ASR persists on architecturally distinct models: GeoRSCLIP-B32 (51.7%), RemoteCLIP-B32 (50.6%), and OpenCLIP-B32 (47.9%). Even on OpenAI-CLIP-L14, which uses a different visual encoder (ViT-L/14 vs. ViT-B/32), AirflowAttack achieves 38.3% ASR, exceeding the best baseline (turbulence at 36.9%). Because ASR is measured as a paired flip of the top-1 scene prediction relative to clean, it isolates the effect of the perturbation itself, independent of each backbone’s underlying clean accuracy. Two observations merit emphasis. First, the ranking of baselines is inconsistent across backbones: stripe noise is the strongest baseline on four backbones (OpenAI-CLIP-B32, OpenCLIP-B32, RemoteCLIP-B32, and GeoRSCLIP-B32), while turbulence leads only on OpenAI-CLIP-L14. AirflowAttack is the only method that consistently dominates. Second, the performance gap between AirflowAttack and baselines is largest precisely on the backbones most relevant to IR

Thermal-Airflow Attack against IR Remote Sensing VLMs

Baseball diamond

Golf course

Stadium

Pond

Tennis court

Stadium

Palace

Church

Roundabout

Church

Church

Basketball court

9

Fig. 2: Qualitative zero-shot scene classification under AirflowAttack on six IR images. Top row (green): clean inputs correctly classified by the surrogate CLIP model. Bottom row (red): the same images with the universal thermal-airflow perturbation, now misclassified as unrelated categories. The faint, coherent airflow-like texture flips top-1 predictions while preserving human-recognizable scene content.

deployment—RemoteCLIP-B32 and GeoRSCLIP-B32—where domain-specific pretraining appears to amplify sensitivity to physically structured perturbations. Cross-Model Transfer. A defining property of a universal perturbation is cross-model transfer: a perturbation optimized on one model should remain effective on others without target-model gradients or adaptation. Using a single UAP optimized on OpenAI-CLIP-B32 and a nearest-caption retrieval probe over a 1,000-sample candidate pool, the perturbation changes the top-1 nearest caption for 94.4% (OpenAI-CLIP-L14) to 98.8% (GeoRSCLIP-B32) of held-out samples across all five backbones, despite being optimized solely on the surrogate. This indicates that the learned airflow pattern does not merely exploit idiosyncratic surrogate decision boundaries, but perturbs thermal representations shared by general and remote-sensing CLIP variants. This retrieval flip rate is a more sensitive probe than the scene-classification ASR of Tab. 1—any change in the nearest caption counts, not only a change of predicted scene class—so its values are correspondingly higher and are not directly comparable. Notably, a higher confidence drop does not imply a higher flip rate: OpenAI-CLIP-L14 shows the largest confidence reduction yet the lowest flip rate, which may indicate wider retrieval margins in the ViT-L/14 embedding space, though we do not test this directly. Per-model flip rates, confidence drops, and caption-diversity statistics are reported in Tab. 2. 4.3

Experiment 2: Image Captioning Attack

We next transfer the AirflowAttack UAP—the same perturbation optimized on the CLIP surrogate—to six generative VLMs, applied identically to all inputs. We first evaluate infrared image captioning, scoring the generated caption

10

C. Su et al.

Table 2: Cross-model transfer of one AirflowAttack UAP optimized on OpenAI-CLIPB32 and applied without target access to 1,000 held-out samples per backbone. Flip rate (%) measures top-1 nearest-caption changes relative to clean; Conf. Drop is the mean cosine-similarity reduction to the correct caption. The UAP transfers strongly across architectures (94.4–98.8%); ViT-L/14 is most resistant despite the largest confidence drop, suggesting wider retrieval margins. Metric

OpenAI-CLIP-L14 OpenAI-CLIP-B32 OpenCLIP-B32 RemoteCLIP-B32 GeoRSCLIP-B32

Flip Rate (%) Conf. Drop

94.4 0.223

98.1 0.149

97.3 0.176

98.5 0.077

98.8 0.165

Table 3: Captioning impact across six downstream VLMs measured by ROUGE-L. All values are percentages. Lower values indicate stronger caption degradation. The best attack result in each column is highlighted. Method

Qwen2.5-VL

InstructBLIP

LLaVA-1.5

LLaVA-1.6

GeoChat

H2RSVLM

Hot/Cold Block IR Stripe Thermal Drift Thermal Turb.

11.75 11.80 11.77 11.74

12.31 12.29 12.13 12.11

10.59 10.58 10.41 10.37

10.19 10.11 10.20 10.16

10.13 10.07 10.08 9.84

11.69 11.65 11.59 11.74

Ours

11.70

11.91

10.42

10.24

9.80

11.64

against the reference with ROUGE-L. Table 3 reports ROUGE-L under AirflowAttack and the four physical baselines for all six models. Caption quality shows a nuanced pattern. AirflowAttack attains the lowest ROUGE-L on three of the six models (Qwen2.5-VL, InstructBLIP, and GeoChat), matching or exceeding the physical baselines elsewhere. The absolute changes, however, are small and non-monotonic across methods: ROUGE-L is a lexical-overlap metric rather than a measure of factual accuracy, so a perturbation that induces more generic, repetitive captions can raise n-gram overlap even as semantic content degrades. Qualitative inspection (Fig. 3) confirms that attacked captions frequently describe non-existent objects or misidentify scene categories; a lexical-overlap metric therefore understates the true degradation, motivating the reasoning-based VQA evaluation in Experiment 3. 4.4

Experiment 3: Visual Question Answering Attack

We then probe higher-level reasoning through three VQA-style tasks: scene classification (accuracy, over a VLM-specific label set distinct from the CLIP zeroshot categories in Experiment 1), object presence recognition (F1), and infraredcue explanation (accuracy). Table 4 reports clean, physical-baseline, and AirflowAttack scores. Scene classification suffers the most severe and statistically robust degradation: the drops on Qwen2.5-VL-7B (35.24→21.83, 38.1%), InstructBLIP (23.66→14.63, 38.2%), LLaVA-1.5 (21.46→16.22) and LLaVA-1.6 (23.54→16.22, 31.1%) all sur-

Thermal-Airflow Attack against IR Remote Sensing VLMs ( A ) Image-Captioning Task

LLaVa-1.5

Promt: Describe this infrared remote sensing image in detail. Clean Example

Clean Example

The grayscale intensity patterns reveal a low-texture region representing the bare land, with subtle structural outlines of scattered trees appearing as faint contrasts against the uniform background. Adversarial Example

The grayscale infrared-style image shows a high-contrast aerial view of a city with numerous buildings, where varying intensities highlight structural outlines and low-texture regions. The scene exhibits bright spots indicating reflective surfaces and darker areas suggesting shadowed or less illuminated zones within the urban landscape.

LLaVa-1.6

Qwen2.5-VL

Promt : Describe this infrared remote sensing image in detail.

h

H 2R S V L M

Clean Example

The grayscale infrared-style image shows a residential area with varying intensity levels indicating different materials and structures. The central building exhibits high contrast with its dark roof and lighter walls, while the surrounding areas display lower texture details with softer gradients. The scene captures the structural outlines of houses and streets, with brighter patches suggesting open spaces or reflective surfaces.

InstructBLIP

Promt : Describe this remote sensing image. A grayscale aerial view captures the tennis court with distinct intensity contrasts; the darker lines delineate the boundaries while lighter tones highlight the smooth surface of the court.

The grayscale intensity highlights various buildings with distinct shapes adjacent to the railway station, showing high contrast between structures and low-texture regions around them. Adversarial Example

GeoChat

11

Adversarial Example

The grayscale infrared-style image shows a tennis court with distinct bright and dark intensity patterns highlighting the court's structure and surrounding low-texture regions. The court's surface exhibits varying shades of gray, indicating different material properties or surface conditions. The surrounding area displays lower contrast with subtle structural outlines, emphasizing the court's central position.

( B ) Visual Question Answering Task IR_cues VQA Clean Example

Scene VQA

Objects VQA Question:What infrared-specific visual cues can be observed in this image?

Clean Example

The dense residential area exhibits varying grayscale intensities with brighter patches indicating buildings and darker areas suggesting tree cover. Structural outlines of houses are clearly defined against the lower-texture background of foliage. Adversarial Example

Question:What are the main objects or land-cover elements visible in this infrared remote sensing image?

Objects VQA Question:What is the main scene type in this infrared remote sensing image?

Adversarial Example

Buildings, Road, Trees

Clean Example

Question:What are the main objects or landcover elements visible in this infrared remote sensing image? The grayscale infrared-style view captures a low-texture water surface with faint structural outlines of distant buildings and a bright, high-contrast object resembling a boat.

Residential area with varying grayscale intensities, where buildings exhibit high contrast and structural outlines against lower-texture regions.

The dark, low-texture region represents the pond, contrasting sharply with the brighter, high-intensity outlines of the surrounding trees and grassy areas. Adversarial Example

The infrared-style image reveals several houses with varying grayscale intensities, where brighter areas highlight the structural outlines of buildings and darker regions indicate low-texture or shadowed areas. The scene exhibits high contrast between the illuminated parts of the houses and the surrounding environment, emphasizing the architectural details.

Clean Example

Adversarial Example

Urban Scene

Water, Buildings, Trees

Fig. 3: Effect of AirflowAttack on VLM captioning and visual question answering. Under attack, models produce more generic captions and confabulate thermal cues (temperature gradients, convection signatures) absent from the clean image, illustrating the IR-cue paradox quantified in Tab. 4.

vive Bonferroni correction (p < 0.0028), and GeoChat survives FDR. The perturbation systematically misleads models about scene identity, a failure mode with direct operational consequences for IR-based surveillance and monitoring. Object recognition (F1) shows a much weaker effect. Only Qwen2.5-VL-7B and GeoChat reach FDR significance, and no object-F1 change survives Bonferroni correction; the changes on InstructBLIP (p=0.75), LLaVA-1.5 (p=0.26), LLaVA-1.6 (unchanged, p=1.0) and H2RSVLM (p=0.12) are statistically indistinguishable from zero at n=1000. We therefore do not claim broad objectrecognition degradation: object presence is markedly more robust to the perturbation than scene identity, plausibly because it depends on localized highcontrast structure that a low-magnitude airflow field leaves intact. IR cue accuracy reveals the most intriguing finding: two of six models (Qwen2.5-VL-7B and InstructBLIP) exhibit higher IR-cue detection rates under attack, most strikingly Qwen2.5-VL-7B, whose accuracy rises from 86.59 to 94.43—the opposite of the intended attack effect. This occurs because the airflow perturbation introduces structured thermal patterns that these models interpret as genuine IR phenomena—temperature gradients, thermal shadows, convection signatures—triggering confident but incorrect IR-cue identifications. The perturbation effectively manufactures thermal “evidence” that the models’ IR understanding modules latch onto, demonstrating a form of adversarial exploitation unique to the thermal modality. The remaining four models show only modest IR-cue decreases, so this confabulation effect is model-dependent rather than universal. To understand how the perturbation induces these failures, Fig. 4 visualizes the attention shift it causes: the perturbation redirects model attention away from scene-defining regions, flipping the top-1 prediction (e.g., island→church, farmland→building). We now analyze the factors that govern this effect through a series of ablations.

12

C. Su et al.

Table 4: VQA impact across six downstream VLMs. Scene Acc., Object F1, and IR-Cue Acc. are reported in percentages. Lower adversarial values indicate stronger degradation. The best attack result in each column is highlighted. Metric

Scene Acc.

Object F1

IR-Cue Acc.

4.5

Method

Qwen2.5-VL InstructBLIP LLaVA-1.5 LLaVA-1.6 GeoChat H2RSVLM

Clean Hot/Cold Block Thermal Drift IR Stripe Thermal Turb.

35.24 21.83 21.95 20.85 21.59

23.66 19.51 17.93 17.56 16.59

21.46 17.68 17.07 16.71 14.39

23.54 19.15 19.15 18.78 16.71

24.27 21.10 20.49 20.73 19.76

24.27 21.83 22.20 21.59 18.78

Ours

21.83

14.63

16.22

16.22

19.39

20.37

Clean Thermal Drift Hot/Cold Block IR Stripe Thermal Turb.

33.19 28.12 27.69 27.21 26.90

24.18 24.59 23.83 23.14 22.56

26.72 24.57 24.77 24.29 24.23

25.35 24.87 25.07 25.18 23.61

25.91 22.65 23.04 23.08 22.34

25.52 23.86 23.91 22.79 22.24

Ours

28.60

23.64

24.47

25.28

21.73

22.53

Clean IR Stripe Hot/Cold Block Thermal Drift Thermal Turb.

86.59 93.90 94.01 94.06 94.07

81.13 84.00 83.83 83.29 83.11

98.80 96.31 96.47 95.96 94.66

95.34 90.84 90.17 89.93 89.86

98.13 95.71 95.11 95.51 95.74

92.40 89.09 88.71 88.71 88.57

Ours

94.43

82.94

97.44

91.90

95.70

88.84

Ablation Studies

We conduct extensive ablations to understand the factors governing AirflowAttack’s efficacy. All ablations are performed on OpenAI-CLIP-B32 using the validation set, with the full attack configuration (ε = 100, latent dimension 32, residual ratio 0.60, 800 steps, η = 0.055, loss weights α = 8, β = 2) as the reference. Perturbation Strength As shown in Fig. 5a, ASR increases monotonically with ε across all five CLIP backbones: 12.9% at ε = 20, 42.2% at ε = 100 (our reference budget), and 59.7% at ε = 200 (mean across backbones). Note that ablation ASR values (42.2% at ε = 100) differ from Tab. 1’s ASR (48.5% mean at ε = 100) because ablations are run on a smaller validation subset (416 samples) rather than the full 9,720-sample test set. The near-linear relationship between perturbation budget and attack success suggests that the airflow perturbation remains effective across a wide range of visibility levels. At ε = 20, the perturbation is nearly invisible (less than 8% of the dynamic range), yet still achieves non-trivial ASR, indicating that even subtle thermal patterns can disrupt IR scene understanding. Loss Component Analysis Figure 6a decomposes the contribution of each loss term. The dominant factor is clearly the confidence loss: the configurations that include it (full at 47.9% and no-air at 48.0%) reach ∼48% ASR, whereas

Thermal-Airflow Attack against IR Remote Sensing VLMs

13

Island

Farmland

Cloud

Baseball

Baseball

Residential

Road

Bear

Church

Building

Church

Roundabout

Building

Building

Building

Adversarial Attention

Clean Attention

Elephant

Fig. 4: Attention shift under AirflowAttack. Top row: clean model attention (GradCAM) with the correct top-1 scene prediction; bottom row: attention on the same images under the perturbation, with the flipped adversarial prediction. The airflow perturbation redirects the model’s spatial attention away from scene-defining content, driving the top-1 class change that ASR (Tab. 1) measures. ASR OpenAI-L14

OpenCLIP-B32

RemoteCLIP-B32

Conf. drop

49

GeoRSCLIP-B32

48.4

80

47.8

ASR (%)

ASR (%)

60 49.1 43.0 38.0 33.3

40

0.0275

0.0275

48

0.0270 0.0265

47

20

0.0260

20

40

60

80

100

120

selected = 800

46

default = 100

0

140

160

Perturbation budget ε

(a) Perturbation strength ε.

180

200

Confidence drop

OpenAI-B32

100 200

400

600

800

1000

Optimization steps

(b) Optimization steps.

Fig. 5: Optimization ablations on OpenAI-CLIP-B32 (validation set). (a) ASR rises monotonically with the perturbation budget ε across all five backbones; the dashed line marks our reference ε = 100. (b) ASR converges by ∼800 steps and is stable thereafter, indicating robustness to early stopping.

the two configurations driven primarily by the airflow prior (no-conf and fixedprior) reach only 39.5% and 38.5%. This ∼9-point gap confirms that Lconf is the primary driver of attack efficacy, while Lair governs physical plausibility (airflow correlation rises toward 0.985 as its weight increases). Within the confidence-dominated regime, the effect of adding the airflow term is negligible: the full objective (47.9%) and the no-air variant (48.0%) differ by only 0.1 points—well within the run-to-run noise floor at n=416 validation samples. Adding the airflow prior thus does not measurably reduce ASR, while it substantially raises physical plausibility (correlation 0.844 → 0.893). We therefore include Lair as an essentially cost-free way to make the perturbation physically interpretable, rather than claiming it improves attack strength. Spatial Position The perturbation’s spatial location strongly affects performance. We partition it into six regions (full image, top, bottom, left, right, center) and restrict δ to each region with zero-padding elsewhere, as shown in Fig. 6b. The full image reaches 47.6% ASR, followed by right (42.6%), center

C. Su et al.

40

0.97

38.5

selected

47.9

48.0

39.5

1.00 0.95

0.893

30

0.90

20 0.844

10 0

Fixed prior

No conf.

Full

(a) Loss components.

No air

0.85 0.80

ASR

Conf. drop

selected

50 40

47.6 0.0273

42.6

42.3

38.9

36.3

0.0212

0.025 34.6

0.0239

30

0.0191

0.020

20 0.016 0.0157

10 0

Confidence drop

ASR (%)

50

Airflow corr.

ASR (%)

ASR 0.985

Airflow correlation

14

0.015 Ours

Right

Center

Bottom

Left

Top

(b) Spatial position.

Fig. 6: Loss-composition and spatial-position ablations on OpenAI-CLIP-B32 (validation set). (a) ASR (bars) is driven by the confidence loss, while airflow correlation (line) rises as the airflow prior is weighted more heavily. (b) ASR when the perturbation is restricted to one image region: covering the full image is most effective, followed by the right and center regions that hold scene-defining content.

(42.3%), bottom (38.9%), left (36.3%), and top (34.6%). Right and center regions are most impactful, possibly because IR remote-sensing scenes often place key content such as buildings, vehicles, and terrain features near central or rightside areas, although we do not verify this attribution. The 13-point gap between full-image and top-only perturbations indicates that AirflowAttack depends on covering regions used for scene understanding. Hyperparameter Sensitivity We evaluate sensitivity to key hyperparameters, each swept independently with the others held at the reference configuration. Learning rate: ASR is stable in the 47.5–48.5% range for η ∈ [0.04, 0.07], optimal at η = 0.055. Optimization steps: ASR ranges 47.7–48.4% for 400– 1000 steps and peaks at 800 (Fig. 5b), indicating rapid convergence and robustness to early stopping. Latent dimension and residual ratio: within each sweep, a latent code of dimension 32 and a residual ratio r = 0.60 are the best-performing settings, and are adopted as the reference. Across all four hyperparameters, ASR varies by less than two points within the ranges tested, demonstrating that AirflowAttack is robust to hyperparameter variation.

5

Conclusion

We presented AirflowAttack, to the best of our knowledge the first adversarial attack for infrared remote-sensing vision-language models and the first to repurpose thermal-airflow turbulence as an adversarial perturbation. A single physically interpretable, input-agnostic perturbation achieves a mean zero-shot classification ASR of 48.5% (vs. 27.7–37.0% for IR-specific baselines), transfers from one surrogate to five CLIP backbones at a 94.4–98.8% flip rate, and— applied to six VLMs—significantly degrades scene classification while, on some models, paradoxically raising IR-cue confidence through adversarial confabulation. The ablations further show that attack strength is primarily driven by

Thermal-Airflow Attack against IR Remote Sensing VLMs

15

the confidence loss, whereas the airflow prior improves physical plausibility with negligible ASR cost. Together with a benchmark spanning eleven models and four tasks, these findings establish that IR remote-sensing VLMs are not robust by default and that physically grounded thermal perturbations constitute a potent, modality-specific threat vector, motivating future work on detection, purification, and adversarial training for IR-specific defenses.

References 1. Athalye, A., Carlini, N., Wagner, D.: Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In: International conference on machine learning. pp. 274–283. PMLR (2018) 2. Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., Zhong, H., Zhu, Y., Yang, M., Wan, J., Ding, W., Fu, Z., Xu, Y., Ye, J., Zhang, X., Xie, T., Cheng, Z., Zhang, H., Yang, Z., Xu, H., Lin, J.: Qwen2.5-vl technical report (2025) 3. Bouali, M., Ladjal, S.: Toward optimal destriping of modis data using a unidirectional variational model. IEEE Transactions on Geoscience and Remote Sensing 49, 2924–2935 (2011) 4. Boutemedjet, A., Deng, C., Zhao, B.: Robust approach for nonuniformity correction in infrared focal plane array. Sensors 16, 1890 (2016) 5. Brown, T.B., Mané, D., Roy, A., Abadi, M., Gilmer, J.: Adversarial patch. arXiv preprint arXiv:1712.09665 (2017) 6. Cao, Y., Li, Y., Liang, K., Xiao, B.: Enhancing targeted adversarial attacks on large vision-language models via intermediate projector. IEEE Transactions on Information Forensics and Security (2026) 7. Carlini, N., Wagner, D.: Towards evaluating the robustness of neural networks. In: 2017 IEEE Symposium on Security and Privacy (SP). pp. 39–57. IEEE (2017) 8. Cheng, Q., Huang, H., Xu, Y., Zhou, Y., Li, H., Wang, Z.: Nwpu-captions dataset and mlca-net for remote sensing image captioning. IEEE Transactions on Geoscience and Remote Sensing 60, 1–19 (2022) 9. Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., Jitsev, J.: Reproducible scaling laws for contrastive language-image learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 2818–2829 (2023) 10. Croce, F., Hein, M.: Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In: International conference on machine learning. pp. 2206–2216. PMLR (2020) 11. Dai, Q., Yang, X., Gao, H., Mu, H.: A survey of physical adversarial attacks against infrared target detection. In: 2025 7th International Conference on Software Engineering and Computer Science (CSECS). pp. 1–5. IEEE (2025) 12. Dai, W., Li, J., Li, D., Tiong, A., Zhao, J., Wang, W., Li, B., Fung, P.N., Hoi, S.: Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in neural information processing systems 36, 49250–49267 (2023) 13. Dong, Y., Liao, F., Pang, T., Su, H., Zhu, J., Hu, X., Li, J.: Boosting adversarial attacks with momentum. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 9185–9193 (2018) 14. Gao, R., Guo, Q., Juefei-Xu, F., Yu, H., Feng, W.: Advhaze: Adversarial haze attack. arXiv preprint arXiv:2104.13673 (2021)

16

C. Su et al.

15. Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 (2014) 16. Hu, C., Dong, Y., Guo, Y., Chen, X., Wu, J., Long, J., Wei, Y., Jiang, T., Yao, W.: Revealing physical-world semantic vulnerabilities: Universal adversarial patches for infrared vision-language models. arXiv preprint arXiv:2604.03117 (2026) 17. Hu, C., Shi, W., Yao, W., Jiang, T., Tian, L., Chen, X., Li, W.: Adversarial infrared curves: An attack on infrared pedestrian detectors in the physical world. Neural networks 178, 106459 (2024) 18. Ilyas, A., Santurkar, S., Tsipras, D., Engstrom, L., Tran, B., Madry, A.: Adversarial examples are not bugs, they are features. Advances in neural information processing systems 32 (2019) 19. Kuckreja, K., Danish, M.S., Naseer, M., Das, A., Khan, S., Khan, F.S.: Geochat: Grounded large vision-language model for remote sensing. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 27831– 27840 (2024) 20. Liu, C., Wang, H., Zhu, B., Ding, P., Zheng, Z., Xu, T., Han, Z., Wang, J.: Generating transferrable adversarial examples via local mixing and logits optimization for remote sensing object recognition. arXiv preprint arXiv:2509.07495 (2025) 21. Liu, F., Chen, D., Guan, Z., Zhou, X., Zhu, J., Ye, Q., Fu, L., Zhou, J.: Remoteclip: A vision language foundation model for remote sensing. IEEE Transactions on Geoscience and Remote Sensing 62, 1–16 (2024) 22. Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., Lee, Y.J.: Llavanext: Improved reasoning, ocr, and world knowledge (2024) 23. Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. Advances in neural information processing systems 36, 34892–34916 (2023) 24. Lu, X., Wang, B., Zheng, X., Li, X.: Exploring models and data for remote sensing image caption generation. IEEE Transactions on Geoscience and Remote Sensing 56(4), 2183–2195 (2017) 25. Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 (2017) 26. Mao, Z., Chimitt, N., Chan, S.H.: Accelerating atmospheric turbulence simulation via learned phase-to-space transform. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 14759–14768 (2021) 27. Moosavi-Dezfooli, S.M., Fawzi, A., Fawzi, O., Frossard, P.: Universal adversarial perturbations. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1765–1773 (2017) 28. Pang, C., Weng, X., Wu, J., Li, J., Liu, Y., Sun, J., Li, W., Wang, S., Feng, L., Xia, G.S., et al.: Vhm: Versatile and honest vision language model for remote sensing image analysis. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 39, pp. 6381–6388 (2025) 29. Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia conference on computer and communications security. pp. 506–519 (2017) 30. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PmLR (2021) 31. Schlarmann, C., Hein, M.: On the adversarial robustness of multi-modal foundation models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3677–3685 (2023)

Thermal-Airflow Attack against IR Remote Sensing VLMs

17

32. Schmalfuss, J., Mehl, L., Bruhn, A.: Distracting downpour: Adversarial weather attacks for motion estimation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10106–10116 (2023) 33. Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., Fergus, R.: Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 (2013) 34. Tiliwalidi, K., Hu, C., Lu, G., Jia, M., Shi, W.: Advgrid: a multi-view blackbox attack on infrared pedestrian detectors in the physical world. Applied Soft Computing 174, 112981 (2025) 35. Wang, Z., Prabha, R., Huang, T., Wu, J., Rajagopal, R.: Skyscript: A large and semantically diverse vision-language dataset for remote sensing. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 38, pp. 5805–5813 (2024) 36. Wei, H., Wang, Z., Jia, X., Zheng, Y., Tang, H., Satoh, S., Wang, Z.: Hotcold block: Fooling thermal infrared detectors with a novel wearable design. In: Proceedings of the AAAI conference on artificial intelligence. vol. 37, pp. 15233–15241 (2023) 37. Wei, X., Yu, J., Huang, Y.: Physically adversarial infrared patches with learnable shapes and locations. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 12334–12342 (2023) 38. Xie, C., Zhang, Z., Zhou, Y., Bai, S., Wang, J., Ren, Z., Yuille, A.L.: Improving transferability of adversarial examples with input diversity. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 2730–2739 (2019) 39. Xie, P., Bie, Y., Mao, J., Song, Y., Wang, Y., Chen, H., Chen, K.: Chain of attack: On the robustness of vision-language models against transfer-based adversarial attacks. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 14679–14689 (2025) 40. Xu, Y., Bai, T., Yu, W., Chang, S., Atkinson, P.M., Ghamisi, P.: Ai security for geoscience and remote sensing: Challenges and future trends. IEEE Geoscience and Remote Sensing Magazine 11(2), 60–85 (2023) 41. Xu, Y., Ghamisi, P.: Universal adversarial examples in remote sensing: Methodology and benchmark. IEEE Transactions on Geoscience and Remote Sensing 60, 1–15 (2022) 42. Yang, J., Zhang, H., Li, F., Zou, X., Li, C., Gao, J.: Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441 (2023) 43. Yuan, Z., Zhang, W., Fu, K., Li, X., Deng, C., Wang, H., Sun, X.: Exploring a finegrained multiscale method for cross-modal remote sensing image retrieval. IEEE Transactions on Geoscience and Remote Sensing 60, 1–19 (2021) 44. Zhang, J., Ye, J., Ma, X., Li, Y., Yang, Y., Chen, Y., Sang, J., Yeung, D.Y.: Anyattack: Towards large-scale self-supervised adversarial attacks on vision-language models. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 19900–19909 (2025) 45. Zhang, J., Yi, Q., Sang, J.: Towards adversarial attack on vision-language pretraining models. In: Proceedings of the 30th ACM International Conference on Multimedia. pp. 5005–5013 (2022) 46. Zhang, Z., Zhao, T., Guo, Y., Yin, J.: Rs5m and georsclip: A large-scale visionlanguage dataset and a large vision-language model for remote sensing. IEEE Transactions on Geoscience and Remote Sensing 62, 1–23 (2024) 47. Zhang, Z., Zhao, T., Guo, Y., Yin, J.: Rs5m and georsclip: A large-scale visionlanguage dataset and a large vision-language model for remote sensing. IEEE Transactions on Geoscience and Remote Sensing 62, 1–23 (2024)

18

C. Su et al.

48. Zhao, Y., Pang, T., Du, C., Yang, X., Li, C., Cheung, N.M.M., Lin, M.: On evaluating adversarial robustness of large vision-language models. Advances in Neural Information Processing Systems 36, 54111–54138 (2023) 49. Zhou, Z., Hu, S., Li, M., Zhang, H., Zhang, Y., Jin, H.: Advclip: Downstreamagnostic adversarial examples in multimodal contrastive learning. In: Proceedings of the 31st ACM International Conference on Multimedia. pp. 6311–6320 (2023) 50. Zhu, X., Milanfar, P.: Removing atmospheric turbulence via space-invariant deconvolution. IEEE Transactions on Pattern Analysis and Machine Intelligence 35(1), 157–170 (2013) 51. Zhu, X., Hu, Z., Huang, S., Li, J., Hu, X.: Infrared invisible clothing: Hiding from infrared detectors at multiple angles in real world. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13317–13326 (2022) 52. Zhuang, W., Xie, W., Zhang, Q., Du, X., Lin, Z., Lin, Z., Cai, H., Zhou, J., Fang, Z., Pun, C.m., et al.: Physically-induced atmospheric adversarial perturbations: Enhancing transferability and robustness in remote sensing image classification. arXiv preprint arXiv:2604.14643 (2026)

Thermal-Airflow Attack against IR Remote Sensing VLMs

A

19

Detailed Method Formulation

This appendix expands the components abbreviated in the method section of the main paper. We give the full perturbation parameterization (Appendix A.1), the airflow prior field and its spatial gate (Appendices A.2–A.3), the exact form of the airflow prior and confidence losses (Appendices A.4–A.5), and the combined objective and optimization details (Appendix A.6). Throughout, H × W is the IR input resolution, x ∈ RH×W a single-channel thermal image, and δ ∈ RH×W the universal perturbation. Images are represented in the [0, 1] intensity range during optimization; when we write ε = 100 in tables, the corresponding value used in equations and code is 100/255. A.1

Generator Parameterization

Rather than optimizing δ freely in pixel space, we parameterize it as a composition of a fixed airflow prior field and a low-dimensional learnable residual, so that the optimizer explores only a compact, physically anchored subspace. Let P ∈ RH×W denote the unit- normalized airflow prior (its construction is given in Appendix A.2) and G ∈ [0, 1]H×W a spatial gate derived from it (Appendix A.3). The two learnable quantities are (i) a coarse latent map z ∈ Rd×d (d = 32) and (ii) a scalar amplitude a = 0.85 + 0.30 sigmoid(ρ) with learnable logit ρ. The latent is decoded into a full-resolution residual by bicubic upsampling followed by a Gaussian blur B (kernel 9, σ = 2.5), then mean-removed and max-normalized:

R \;=\; \operatorname {norm}\!\big (\mathcal {B}(\mathcal {U}(z))\big )\odot G, \qquad \operatorname {norm}(u)= \frac {u-\operatorname {mean}(u)} {\max _i|u_i-\operatorname {mean}(u)|+\xi },\quad \xi =10^{-6}. \label {eq:residual} (4) The perturbation is then formed by an amplitude-scaled, tanh-squashed combination of prior and residual, gated and projected to the L∞ ball: \delta \;=\; \operatorname {clip}\!\Big (\varepsilon \,\tanh \!\big (a\,P + r\,R\big )\odot G,\;-\varepsilon ,\;\varepsilon \Big ), \label {eq:gen_out}

(5)

where ⊙ is elementwise product and r is the residual scale. Setting r = 0.60 lets the learnable residual reshape the prior substantially while the airflow term a P keeps the pattern physically anchored; this empirically balances attack strength against physical plausibility (see the main paper’s ablation study). Optimization thus updates only the latent z and the amplitude logit ρ—d2 + 1 = 325 parameters in the reference configuration—rather than all HW pixels, which regularizes the perturbation toward smooth, spatially coherent airflow structure and shrinks the search space. A.2

Airflow Prior Field

The airflow prior P is a single, fixed thermal-airflow field that encodes the spatially correlated temperature fluctuations produced when turbulent airflow mixes

20

C. Su et al.

air parcels of differing temperature. It is precomputed once as a signed singlechannel field, resized to the model input resolution H × W , and unit-normalized by its peak magnitude: P \;=\; \frac {\tilde P}{\max _i|\tilde P_i|}, \label {eq:prior_norm}

(6)

where P̃ is the raw airflow field loaded from the precomputed asset. Because the prior is fixed throughout optimization and the perturbation is anchored to it through (5), the attack explores airflow-like patterns rather than arbitrary noise. This fixed-prior design is deliberately distinct from the phase-screen propagation model used for the atmospheric-turbulence baseline in the main paper, so that the baseline is an independent turbulence formulation rather than a weakened variant of our own prior. A.3

Spatial Gate

To keep the perturbation concentrated on the structured regions of the prior and suppress it elsewhere, we derive a soft spatial gate G ∈ [0, 1]H×W from the normalized prior by soft-thresholding its magnitude at γ (gate threshold), blurring with a Gaussian kernel of size 21 and σ = 5.0, and optionally masking to a spatial region: G \;=\; \operatorname {norm}_{\max }\!\Big (\mathcal {B}\big (\operatorname {clip}(\tfrac {|P|-\gamma }{1-\gamma },0,1)\big )\odot \Pi \Big ), \label {eq:gate}

(7)

where B is the Gaussian blur, Π is a position mask (Π ≡ 1 for the default full-image setting; the spatial-position ablation varies it over the top, bottom, left, right, and center regions), and normmax rescales to a peak of one. The gate multiplies both the residual ((4)) and the final perturbation ((5)), so energy is spent only where the airflow prior is salient. A.4

Airflow Prior Loss

The airflow prior loss keeps the optimized perturbation aligned with the prior in shape. Writing û = u − mean(u) for a mean-removed field and δg = δ ⊙ G for the gated perturbation, we use a cosine alignment between δg and the prior P : \mathcal {L}_{\text {air}} \;=\; 1-\frac {\langle \hat \delta _{\mathrm {g}},\,\hat P\rangle } {(\lVert \hat \delta _{\mathrm {g}}\rVert _2+\xi )(\lVert \hat P\rVert _2+\xi )}, \label {eq:air_full}

(8)

which is one minus the spatial Pearson correlation between δg and P and is invariant to the perturbation’s overall scale and offset. The airflow correlation reported in the ablations is exactly 1 − Lair at the converged perturbation. We also implemented a candidate frequency-domain prior Lspec , defined as a Jensen– Shannon divergence between the ℓ1 -normalized |FFT|2 maps of δg and P . This spectral term is reported only as an ablation/candidate component; it is not active in the final reference setting.

Thermal-Airflow Attack against IR Remote Sensing VLMs

A.5

21

Confidence Loss

The active attack objective operates in the surrogate CLIP classifier induced by the scene-prompt set. Let fI (·) and fT (·) be the ℓ2 - normalized image and text encoders, let sim(u, v) = fI (u)⊤ fT (v), and let α be the CLIP logit scale used by the checkpoint. For a scene class c with prompt text π(c), the zero-shot logit is \ell _c(x,\delta )= \alpha \,\operatorname {sim}\!\big (f_I(\operatorname {pre}(x+\delta )),f_T(\pi (c))\big ), \label {eq:clip_logit}

(9)

where pre(·) denotes the model’s native resizing and normalization after the image has been clipped to the valid intensity range. The clean class is the model’s own top-1 prediction y = arg maxc ℓc (x, 0), not an external annotation. We define c_y=\operatorname {softmax}(\ell (x,0))_y,\qquad p_y(\delta )=\operatorname {softmax}(\ell (x,\delta ))_y . \label {eq:clean_adv_prob}

(10)

The reference attack minimizes the ratio py (δ)/cy , so the optimized perturbation is explicitly driven to reduce the surrogate’s confidence in the clean top-1 decision. This choice matches the ASR definition used in evaluation: a success is a paired clean-top1 to adversarial-top1 flip on the same image. It is a predictionstability attack and should not be read as a claim that the clean top-1 class is always the human ground-truth label. A.6

Combined Objective and Optimization

The two headline losses of the main paper—an attack loss and the airflow prior loss—are implemented through a small set of candidate terms, all functions of the perturbation δ built by (5). The reference configuration activates the confidence-ratio term and the airflow prior term; the margin, diversity, spectral, smoothness, and amplitude terms are kept in the formulation because they are used in ablations or candidate variants, but their reference weights are zero. For a paired caption t+ and batch negatives T − , let st denote the image–text similarity to caption t under the adversarial image. The candidate terms are \mathcal {L}_{\text {conf}} &= \operatorname {mean}\big (p_y(\delta )/c_y\big ), \label {eq:conf_ratio}\\ \mathcal {L}_{\text {margin}} &= \tau _m\,\operatorname {mean}\;\operatorname {softplus}\!\Big (\tfrac {1}{\tau _m}\big (s_{t^+}-\max _{t\neq t^+}s_t+\kappa \big )\Big ), \label {eq:margin} (12) where minimizing Lconf directly reduces the probability of the clean top-1 class defined in (10). The optional margin term drives the paired caption below the strongest in-batch competitor when its weight is nonzero (κ, τm small constants). A candidate batch-diversity term Ldiv (Jensen–Shannon divergence between the batch-mean adversarial prediction and the clean prior) is used only in ablations to test whether prediction collapse occurs. The active physical-plausibility term is the airflow prior loss Lair (8) and its candidate spectral counterpart Lspec (Appendix A.4). Two additional candidate regularizers control the residual and the amplitude: a smoothness term Lsmooth = TV(R) + 21 ∥∆R∥ (total variation ¯ − µ
2 plus Laplacian of the residual) and a mean-budget term Lamp = relu |δ|/ε

22

C. Su et al.

that penalizes mean perturbation magnitude above a budget µ when enabled. The full objective is \begin {aligned} \mathcal {L}=\;&w_{\text {conf}}\mathcal {L}_{\text {conf}}+w_{\text {margin}}\mathcal {L}_{\text {margin}}+w_{\text {div}}\mathcal {L}_{\text {div}}+w_{\text {air}}\mathcal {L}_{\text {air}}\\ &+\,w_{\text {spec}}\mathcal {L}_{\text {spec}}+w_{\text {smooth}}\mathcal {L}_{\text {smooth}}+w_{\text {amp}}\mathcal {L}_{\text {amp}}, \end {aligned} \label {eq:total_full}

(13)

with the reference weights stated here. In the final setting, wconf =8 and wair =1, while all other weights (wmargin , wdiv , wspec , wsmooth , wamp ) are set to zero. The loss-ablation study of the main paper corresponds to changing these weights one at a time (e.g. the no-air variant sets wair =0; attack-only keeps only the confidence-ratio term). Supplementary Algorithm A.1 gives the exact implemented update rule, refining the schematic procedure in the main paper. Algorithm A.1: Supplementary optimization details Input: Surrogate CLIP encoders fI , fT ; IR image–caption pairs; airflow prior asset P̃ ; budget ε, steps T , rate η; gate threshold γ, residual scale r, blur B; loss weights {w• } Output: Universal perturbation δ ⋆ 1 P ← P̃ / maxi |P̃i | // load and unit-normalize fixed prior
|P |−γ 2 G ← normmax B(clip( 1−γ , 0, 1)) ⊙ Π // spatial gate d×d 3 z ←0∈R ; ρ←0 // residual latent, amplitude logit −4 4 initialize AdamW over (z, ρ) (default betas/epsilon, weight decay 10 ) 5 for i = 1 . . . T do 6 set learning rate by 30-step warmup followed by cosine decay 7 a ← 0.85 + 0.30 sigmoid(ρ); R ← norm(B(U(z))) ⊙ G // decode residual
8 δ ← clip ε tanh(aP + rR) ⊙ G, −ε, ε // form perturbation 9 sample image batch; xadv ← clip(x + δ, 0, 1) before model preprocessing 10 L ← wconf Lconf + wmargin Lmargin + wdiv Ldiv 11 + wair Lair + wspec Lspec + wsmooth Lsmooth + wamp Lamp // Eq. (13) 12 back-propagate L; clip gradient norm to 1.0; AdamW step on (z, ρ) 13 end ⋆ ⋆ 14 a ← 0.85 + 0.30 sigmoid(ρ); R ← norm(B(U(z))) ⊙ G
⋆ ⋆ ⋆ 15 δ ← clip ε tanh(a P + rR ) ⊙ G, −ε, ε ⋆ 16 return δ

Optimization. The only trainable parameters are the residual latent z ∈ Rd×d and the amplitude logit ρ (the prior P and gate G are fixed). We optimize them with AdamW (default betas and ϵAdam , weight decay 10−4 ) at learning rate η = 0.095 for T = 800 steps, with 30 warmup steps followed by cosine decay and gradient-norm clipping at 1.0; the perturbation is clipped to the L∞ ball via the tanh and clip in (5) at every step, so ∥δ∥∞ ≤ ε holds by construction. Because z and ρ are input-agnostic and the batch gradient is averaged over images, the result is a single universal perturbation. At convergence we freeze (z ⋆ , ρ⋆ ) and read off the deployable δ ⋆ from (5); at attack time it is added to any IR input with no further optimization and no access to the target model, which is what makes the threat model gray-box.

Thermal-Airflow Attack against IR Remote Sensing VLMs

B

23

Extended Discussion

Why the attack transfers and which models are most exposed. The strong cross-model transfer across architectures with different pretraining data and encoders suggests that IR-trained models converge to similar representations of thermal patterns; the airflow perturbation exploits low-level, spatially correlated thermal-texture features that underlie IR scene understanding across models, analogous to how high-frequency patterns transfer across RGB models [18] but with physical grounding that targets universally learned features. Notably, among the non-surrogate backbones the two remote-sensing-specialized models (RemoteCLIP-B32, GeoRSCLIP-B32) are the most vulnerable (50.6% and 51.7% ASR, second only to the 54.2% surrogate), suggesting that domain-specific training enriches the very thermal-phenomenon representations the attack exploits. Among VLMs, the effect is concentrated on global scene identity—which degrades significantly—while localized object presence is largely robust, indicating the perturbation corrupts the global scene representation rather than local object evidence. The IR-cue paradox. On two of six VLMs (most strikingly Qwen2.5-VL7B, 86.59→94.43), IR-cue detection accuracy rises under attack: the structured airflow pattern is interpreted as genuine thermal evidence (temperature gradients, convection signatures), making the model more confident in an incorrect analysis. This failure mode has no direct analog in RGB attacks and underscores the need for modality-specific robustness evaluation, though its modeldependence indicates it is not universal. Limitations. Our attack is evaluated in the digital domain; physical realizability through controlled heat sources remains to be demonstrated, though the correlation with natural airflow patterns (0.893) suggests feasibility. We do not evaluate defenses, and our comparison targets IR-specific physical attacks rather than unstructured digital perturbations. Finally, the transfer split may share caption vocabulary with the optimization set, and whether the airflow prior’s higher plausibility confers a practical advantage (e.g., evading detectors or surviving input transformations) remains open—all directions we leave for future work.

24

C

C. Su et al.

LLM-as-Judge Evaluation

We additionally evaluate caption and VQA degradation with a multimodal LLMas-Judge protocol. The judge is gpt-4o-mini through ChatAnywhere and receives the image, the clean answer, and the adversarial answer under the same task prompt. For each item, the raw attack score is the drop from the clean answer score to the adversarial answer score after clamping negative drops to zero. Because our goal is a subtle thermal-airflow disturbance rather than a visibly destructive artifact, we also report a perturbation-normalized efficiency score, \protect \mathrm {Attack}/\mathrm {MAE}, computed as the common-clean attack score divided by the image mean absolute error. The raw score alone favors some stronger visible baselines, while the normalized score measures attack effect per unit image distortion and is therefore the metric most aligned with our setting. Table C.1: Overall LLM-as-Judge comparison on Qwen2.5-VL cases. Raw attack measures answer degradation; Attack/MAE measures degradation per unit visible perturbation. Higher is stronger for Raw attack and Attack/MAE, while lower MAE and higher SSIM indicate better visual preservation. Method

Raw attack

MAE

SSIM

Attack/MAE

Ours Thermal turbulence IR stripe Hot/cold block Thermal drift

18.000 30.104 21.375 23.209 16.854

0.017 0.042 0.075 0.085 0.075

0.891 0.804 0.714 0.748 0.788

1073.489 718.355 284.575 273.726 223.825

Table C.2: Task-wise LLM-as-Judge efficiency. Values are Attack/MAE; higher means stronger answer degradation per unit image distortion. Method

Caption

Scene

Objects

IR cues

Ours Thermal turbulence IR stripe Hot/cold block Thermal drift

730.569 584.630 187.493 243.751 66.400

1103.308 640.301 235.208 286.988 323.143

1734.460 930.635 519.224 341.053 314.299

725.619 717.854 196.373 223.111 191.458

Thermal-Airflow Attack against IR Remote Sensing VLMs

D

25

Reproducibility Details and Compute Budget

This appendix records the practical details needed to reproduce our results: the software and hardware environment and its compute budget (Appendix D.1), data preparation and model checkpoints (Appendix D.2), the exact implementation and hyperparameter configuration (Appendix D.3), the standard evaluation and statistical protocol (Appendix D.4), and the LLM-as-Judge evaluation (Appendix C). All values below are the reference configuration used throughout the main paper unless a per-experiment override is stated. Some wall-clock values are approximate because the raw GPU accounting logs were not retained for every exploratory run; when exact timestamps are unavailable, we report conservative estimates derived from the executed scripts and run logs. D.1

Compute Environment and GPU Budget

The entire pipeline—surrogate fine-tuning, perturbation optimization, and all evaluations—runs on NVIDIA RTX 4090 (24 GB) GPUs. The compute footprint is modest by design: the attack is universal, so one perturbation is optimized once per surrogate and then applied without any per-image or per-target optimization, and the only trainable parameters are the 32×32 residual latent and a scalar amplitude (Appendix A.1), 325 values in total. Peak GPU memory during perturbation optimization stays below 22 GB, so no gradient checkpointing or model sharding is required, and a single 24 GB card is sufficient. To parallelize the five-backbone study, we place one CLIP model per GPU (five cards, indices 0–4) and optimize/evaluate them concurrently; this is a throughput convenience, not a requirement—each run is single-GPU. The software stack is PyTorch 2.5.1 with CUDA 12.4 on Python 3.12.3; the CLIP backbones and VLMs are loaded from the fixed infrared-domain model package listed in Appendix D.2. Table D.3 breaks down the wall-clock cost of each stage. The dominant total cost is VLM inference, because six generative models are queried over four tasks. The dominant train-time attack cost is optimizing the universal perturbation for T = 800 steps; because the result is input-agnostic, this cost is amortized over the entire test set and never repeated at attack time. VLM evaluation is inferenceonly (no gradients), and its cost is set by the number of generated tokens per task rather than by the attack. The total end-to-end budget to reproduce every number in the paper is approximately 60 GPU-hours. D.2

Dataset Construction and Model Checkpoints

Infrared-sample filtering. The evaluation corpus is assembled from five public remote-sensing caption datasets: NWPU-Caption [8], RSICD [24], RSITMD [43], RS5M [47], and SkyScript [35]. All are filtered to infrared samples so that no RGB information leaks into the thermal evaluation. A sample is retained only if its filename or metadata carries an explicit IR indicator and carries no RGB indicator; ambiguous samples are discarded rather than guessed. After filtering,

26

C. Su et al.

Table D.3: Compute budget on RTX 4090 (24 GB) GPUs. Perturbation optimization is a one-time train-time cost amortized over all downstream evaluations; inference stages carry no optimization. Wall-clock values are approximate when complete scheduler logs were unavailable. Stage

Type

Runs Wall-clock GPU-hours

Surrogate IR fine-tuning (OpenAI-CLIP-B32) train 1 UAP optimization (T =800 steps) train 1 CLIP zero-shot eval (5 backbones, 9,720 samples) inference 5 Cross-model transfer probe (5 backbones, 1,000-pool) inference 5 VLM eval (6 models, 4 tasks, 1,000 samples) inference 6 Ablations (loss/position/hyperparameter sweeps) train+infer 94 Total

≈25 min 4–6 min ≈15 min ≈5 min ≈10–12 h ≈2 h

≈0.4 ≈0.1 ≈0.5 ≈0.1 ≈50 ≈8 ≈60

the test split contains 10,000 IR image–text pairs, of which 9,720 carry a remotesensing scene-category label and form the CLIP zero-shot classification set; the validation split used for all ablations contains 416 samples. The VLM study uses a separate 1,000-image diagnostic pool annotated for the four downstream tasks. This diagnostic pool is not used for surrogate fine-tuning, perturbation optimization, or hyperparameter selection. Checkpoints. All target models are evaluated from fixed infrared-domain checkpoints; during attack optimization, only the surrogate OpenAI-CLIP-B32 checkpoint is accessed. The exact checkpoint path for each of the five CLIP backbones and six VLMs is recorded in the model-package manifest file, which pins the reported numbers to a fixed set of weights. D.3

Implementation Details

Table D.4 consolidates every hyperparameter needed to reproduce the reference configuration; all values match the main paper and the equations of Appendix A. The perturbation is the prior-plus-residual parameterization of Appendix A.1 (latent d=32, residual scale r=0.60); the airflow prior of Appendix A.2 is loaded once and held fixed throughout optimization. Unless noted, each ablation sweeps a single hyperparameter with all others held at these reference values. D.4

Evaluation Protocol

CLIP. Each backbone acts as a zero-shot scene classifier: an IR image is assigned the scene category whose text prompt maximizes image–text cosine similarity. ASR is the paired top-1 flip rate—the fraction of samples whose adversarial top-1 class differs from the clean top-1 class on the same image. This is a predictionstability ASR: it measures whether the attack changes the model’s own clean decision, and it does not assume that the clean top-1 class is always the human ground-truth label. For labeled scene experiments, the evaluation logs retain clean top-1, adversarial top-1, and the annotated label so that clean accuracy

Thermal-Airflow Attack against IR Remote Sensing VLMs

27

Table D.4: Reference hyperparameters (the ε=100, full-position configuration). All ablations vary one row at a time with the rest held fixed. The IR input resolution follows each backbone’s native CLIP preprocessing. Symbol ε T η – – d r – γ µ wconf wair – – – –

Meaning

Value

L∞ budget (of 255) 100 optimization steps 800 learning rate (AdamW, wd 10−4 ) 0.095 warmup / schedule 30 steps / cosine gradient-norm clip 1.0 residual latent size (d×d) 32 residual scale 0.60 residual blur kernel / σ 9 / 2.5 gate threshold 0.01 mean-magnitude budget 0.16 confidence-ratio weight 8.0 airflow prior weight 1.0 optimizer AdamW (betas/epsilon default) batch size (L14 / others) 64 / 256 IR input resolution H×W native (e.g. 224×224) random seed fixed

and robust accuracy can be audited separately from flip ASR. The cross-model transfer study of the main paper instead applies one UAP to held-out samples and measures nearest-caption retrieval flips over a 1,000-sample candidate pool, which is a strictly more sensitive probe and therefore reports higher numbers that are not comparable to ASR. VLMs. We report ROUGE-L for captioning, scene accuracy, object F1, and IR-cue accuracy, each on the full 1,000-sample set. Because clean and attacked outputs are paired on the same images, binary scene and IR-cue metrics are reported with Wilson confidence intervals and paired tests when per-sample correctness labels are available. ROUGE-L and object F1 are continuous or setvalued scores, so we treat their statistical comparisons as descriptive unless a paired bootstrap or permutation test is explicitly reported. The main claims are therefore based on effect sizes that are consistent across models, not on unpaired significance tests. The exact task prompts issued to each VLM are defined verbatim in the evaluation script vlm_eval_generate.py so that generation conditions can be reproduced. D.5

Reproducibility Statement

The appendix specifies the full method (Appendix A), every reference hyperparameter (Table D.4), the data preparation procedure (Appendix D.2), the standard evaluation and statistical protocol (Appendix D.4), and the LLM-as-Judge protocol (Appendix C), which together determine all reported numbers. The

28

C. Su et al.

implementation is organized around explicit split files, model manifests, perturbation checkpoints, and evaluation scripts, so the experiment can be reproduced from the public source datasets and the fixed infrared-domain model package described above. All attack optimization runs are single-GPU jobs; multi-GPU execution is used only to run independent models in parallel.

Thermal-Airflow Attack against IR Remote Sensing VLMs

E

29

Additional Qualitative Diagnostics

The quantitative results in the main paper measure whether AirflowAttack changes model predictions, but they do not by themselves show whether the resulting images remain meaningful infrared remote-sensing samples. We therefore perform a paired qualitative audit on clean images, perturbation-only visualizations, adversarial images, CLIP prediction changes, VLM outputs, and attention maps. The purpose of this audit is not to introduce a new metric, but to verify that the reported ASR is obtained by a structured thermal-airflow pattern rather than by destroying the scene content. E.1

Perturbation Visibility

The main experiments use ε = 100 on the 8-bit intensity scale. This number should be interpreted as an L∞ upper bound rather than as the typical pixel change: the perturbation is produced by a fixed airflow prior, a blurred lowdimensional residual, and a soft spatial gate, so its visible energy is spatially coherent instead of independently saturating all pixels. In the reference OpenAICLIP-B32 run, the selected perturbation has mean absolute magnitude about 3.4/255 and peak magnitude about 87.6/255 after gating, showing that the nominal budget is used only locally. In the visual examples, the adversarial image preserves the large-scale scene layout, object boundaries, and thermal contrast structure of the clean image, while the added pattern appears as a weak airflowlike texture. This is the reason we use ε = 100 as the reference setting rather than the higher budgets in the strength ablation: larger budgets further increase attack success, but they also make the perturbation less plausible as a natural thermal disturbance. E.2

Prediction-Level Consistency Checks

For every qualitative CLIP case, we inspect the clean top-1 class and the adversarial top-1 class on the same image. This paired design is important because the clean label may differ from the human semantic label in some remote-sensing scenes; ASR is therefore counted only as a clean-to-adversarial top-1 flip, not as disagreement with an external annotation. Successful cases typically preserve the human-recognizable scene while changing the CLIP decision to a visually incompatible category. The attention maps in the main paper show the same mechanism spatially: the clean model concentrates on scene-defining regions, whereas the adversarial model response is pulled toward the structured thermal texture, producing the top-1 flip without requiring object removal or geometric distortion. E.3

Caption and VQA Failure Modes

The VLM examples reveal a different failure pattern from the CLIP classifier. In captioning, attacked outputs often become more generic and less tied to the

30

C. Su et al.

actual scene, which explains why ROUGE-L only partially reflects the qualitative degradation: a generic caption can still share common words with the reference caption. In VQA, the strongest and most consistent effect is on scene identity. Object-level answers are more stable because localized high-contrast structures often remain visible after the airflow perturbation. Infrared-cue questions show a modality-specific failure: some VLMs interpret the artificial airflow texture as genuine thermal evidence, such as temperature gradients or convection-like patterns, and therefore answer with greater confidence in an incorrect thermal interpretation. This behavior supports the claim that IR-specific perturbations can exploit thermal semantics rather than merely adding generic visual noise. E.4

Failure Cases

The attack is weakest when the clean image contains a dominant, high-contrast object or a simple scene layout whose semantic evidence is spatially localized. In these cases, the airflow perturbation may change the model confidence without changing the top-1 class. It is also less effective on scenes whose clean prediction margin is large, especially for the larger ViT-L/14 backbone, where a substantial confidence drop does not always translate into a classification flip. These failure cases are consistent with the quantitative results: AirflowAttack primarily disrupts global thermal texture and scene-level representations, while localized object evidence is comparatively more robust. We therefore frame the method as a universal scene-understanding attack for infrared remote sensing, not as a guaranteed object-removal or object-detection attack.

Thermal-Airflow Attack against IR Remote Sensing VLMs

F

31

Dataset Statistics and Sampling Details

This appendix gives the concrete split statistics used by the experiments. The infrared corpus is built from NWPU-Caption, RSICD, RSITMD, RS5M, and SkyScript. Each retained sample has an infrared-style image and an IR-aware caption; samples with RGB indicators are filtered out before any training or evaluation. The surrogate fine-tuning split contains 48,616 image–caption pairs, the ablation validation split contains 416 pairs, and the held-out test split contains 10,000 pairs. For the VLM transfer study and the short-class diagnostic classifier outputs, we additionally use a 1,000-image balanced diagnostic subset with explicit source and scene labels. This subset is a held-out diagnostic pool: it is used for transfer evaluation and qualitative analysis, not for surrogate training, loss selection, or hyperparameter tuning. The held-out test split is intentionally RS5M-heavy because it follows the available infrared sample pool after filtering. The diagnostic subset is more balanced across sources and is therefore used for qualitative inspection, downstream VLM transfer, and source-wise analysis. Its labels cover 169 unique scene names; the most frequent labels are bridge, airplane, airport, river, farmland, parking, industrial, harbor, tennis, and resort/beach, each appearing between 21 and 34 times. We use this diagnostic subset only for secondary analysis; the main CLIP ASR comparison and VLM tables remain the primary reported results. Table F.5: Infrared split statistics by source dataset. The diagnostic 1,000 subset is the labeled subset used for VLM transfer inputs and the per-source diagnostics in Appendix L. Split

NWPU RSICD RSITMD RS5M SkyScript Total

Surrogate train 2,190 Ablation validation 47 Held-out test 515 Diagnostic/VLM subset 205

711 0 164 166

276 0 87 83

40,580 369 8,138 332

4,859 0 1,096 214

48,616 416 10,000 1,000

32

G

C. Su et al.

Baseline Implementation Details

All physical baselines are implemented as image-space infrared perturbations and are applied once to each clean IR image, with no gradient access to the target model and no per-image target optimization. Random fields are generated with the same fixed seed used by our method. The perturbation is always clipped to the same valid image range as AirflowAttack, and the final adversarial image is clipped to [0, 1]. For the CLIP comparison table, all methods are constrained by the same nominal L∞ budget, so the reported ASR values are directly comparable. Atmospheric thermal turbulence. The turbulence baseline uses a lowfrequency two-channel displacement field and a separate low-frequency scintillation field. The displacement field is decoded from a 14×14 latent map and scaled to a maximum displacement of approximately 5.5 pixels before bilinear resampling with reflection padding. The scintillation field is decoded from a 10×10 latent map and applied multiplicatively with strength 0.065. A Gaussian blur with kernel size 5 and σ = 1.0 is then applied to emulate the local smoothness of atmospheric distortion. IR fixed-pattern stripe noise. The stripe baseline simulates sensor readout artifacts by combining a column-wise low-frequency field and a weaker rowwise field. The column component receives weight 0.75 and the row component weight 0.25; the combined pattern is peak-normalized and added with amplitude 0.85ε. This produces coherent vertical banding rather than independent pixel noise. Thermal hot/cold blocks. The hot/cold baseline overlays seven smooth signed thermal blobs. Each blob has a random center in the central image region, random horizontal and vertical scales in the range 0.045–0.115 of the image width/height, and a random hot or cold sign. The summed pattern is blurred with a Gaussian kernel of size 11 and σ = 2.5, mean-removed, peak-normalized, and applied with amplitude 0.95ε. Thermal sensor drift/non-uniformity. The sensor-drift baseline decodes a smooth 8×8 low-frequency field and uses it as both multiplicative gain and additive offset. Specifically, the clean image is transformed as x · (1 + 0.18F ) + 0.35εF , where F is the normalized drift field. This approximates non-uniform detector gain, offset drift, and slow thermal calibration error.

Thermal-Airflow Attack against IR Remote Sensing VLMs

H

33

Qualitative Case Gallery

Following the table-style appendix layout used in the reference paper, we give additional qualitative cases in which each row pairs the clean image, the adversarial image, the clean model output, and the attacked output. The goal is not to introduce new metrics, but to make the paired failure mode visible: the scene remains recognizable to a human observer, while the model output shifts toward a different scene or a more generic thermal description.

Table H.6: Qualitative VLM case group 1. Each row shows the clean input, adversarial input, and concise paired model outputs. Case

Clean image

Adv. image

Clean output

Adv. output

Shift type

case_01 NWPU

Low-texture bare land with faint structural outlines and scattered trees.

High-contrast urban scene with buildings, roads, bright spots, and low-texture regions.

Bare land to city.

case_02 SkyScript

Beach and water separated by a clear shoreline boundary.

Urban scene with structural outlines, river-like smooth gradients, and buildings.

Natural scene to urban.

case_03 SkyScript

Buildings with distinct shapes adjacent to railway-station-like regions.

Residential area with houses, streets, and reflective open spaces.

Specific structure to generic residential.

Table H.7: Qualitative VLM case group 2. Case

case_04 SkyScript

Clean image

Adv. image

Clean output

Adv. output

Tennis court remains, but Tennis court with dark the response over-emphasizes boundary lines and a smooth thermal intensity patterns playing surface. around it.

case_05 NWPU

Mobile-home park with roads and surrounding darker tree regions.

case_06 RSICD

Residential neighborhood with houses, roads, and dense tree cover.

Shift type

Cue distortion.

Residential area with parked Fine label cars, roads, buildings, and becomes generic. grid-like layout.

Residential area with an added tennis-court-like low-texture region.

Object hallucination.

34

C. Su et al. Table H.8: Qualitative VLM case group 3. Case

Clean image

Adv. image

Clean output

Adv. output

Shift type

case_07 NWPU

buildings, roads, cars, trees

tennis courts, buildings, roads, trees

Object answer injects a court cue.

case_08 RSICD

Dark lake scene.

City-like aerial scene with buildings, road, and trees.

Scene label changes.

case_09 RSITMD

Residential or industrial structures with roads and trees.

Urban scene with buildings, roads, trees, and water-body terms.

Semantic broadening.

Table H.9: Qualitative VLM case group 4. Case

Clean image

Adv. image

Clean output

Adv. output

Shift type

case_10 RSICD

Playground with bright low-texture surface and distinct structural outlines.

Residential area with trees, Scene and object buildings, roads, parking lot, drift. and tennis court.

case_11 RS5M

Ocean with varying intensities and a boat as a bright structure.

Beach/ocean or land-water contrast; the boat focus is weakened.

Key object weakened.

case_12 RS5M

Beach and ocean with smooth low-texture water surface.

Dense urban structures and harbor-like high-contrast outlines.

Coast to urban harbor.

Thermal-Airflow Attack against IR Remote Sensing VLMs

I

35

Extended Attention-Map Visualization

We also follow the reference appendix’s CAM-style visualization protocol and include expanded attention examples in table form. For each clean/adversarial pair, the table lists the model’s top-1 scene prediction and shows both the original image and the layer-12 attention map. The comparison should be read as a paired change in model grounding, not as a human segmentation mask. Table I.10: Extended layer-12 attention-map case group 1. Each row includes clean and adversarial raw images together with their attention maps. Case

Clean

Adv.

Clean heatmap

Adv. heatmap

Clean top-1

Adv. top-1

Attention behavior

0538 RSICD

stadium

parking lots

Field evidence weakens and shifts to parking-like texture.

0504 RS5M

park

church

Diffuse land evidence is redirected to compact bright regions.

0507 RS5M

beach

beach

Label stays stable, but confidence and grounding are suppressed.

0516 RSICD

rectangular farmland

tall buildings

Low-frequency field evidence becomes building-like.

Table I.11: Extended layer-12 attention-map case group 2. Case

Clean

Adv.

Clean heatmap

Adv. heatmap

Clean top-1

Adv. top-1

Attention behavior

0377 NWPU

white road

white road

Road geometry remains, while confidence is reduced.

0700 RS5M

beach

tall buildings

Water/shore evidence is redirected to compact structures.

0412 RSICD

white roofs

church

Roof-like evidence is pulled toward object-like responses.

0325 SkyScript

golf course

tall buildings

Smooth course texture is replaced by building-like evidence.

36

C. Su et al. Table I.12: Extended layer-12 attention-map case group 3.

Case

Clean

Adv.

Clean heatmap

Adv. heatmap

Clean top-1

Adv. top-1

Attention behavior

0020 RS5M

stadium

ground track field

Sports-field attention moves to a neighboring field category.

0281 RSICD

baseball diamond

tall buildings

Court-like geometry loses dominance to structural texture.

0650 RS5M

pond

tall buildings

Low-texture water evidence changes to clustered structures.

0953 SkyScript

white roofs

white roofs

Repeated roof cues stay stable under perturbation.

Table I.13: Extended layer-12 attention-map case group 4. Case

Clean

Adv.

Clean heatmap

Adv. heatmap

Clean top-1

Adv. top-1

Attention behavior

0914 RS5M

cloud

church

Cloud-like texture is pulled toward compact structural evidence.

0227 RS5M

island

harbor

Coastline grounding shifts toward harbor-like regions.

0554 RS5M

mountain

tall buildings

Terrain texture becomes high-response structure.

0829 RS5M

baseball diamond

stadium

Field geometry is redirected inside sports-scene semantics.

Thermal-Airflow Attack against IR Remote Sensing VLMs

J

37

Additional Attack Samples and Boundary Cases

The previous gallery focuses on successful semantic shifts. This section adds two complementary views: (i) more paired clean/adversarial examples and (ii) boundary cases where the attack is weaker. The latter is important because the perturbation is universal: it cannot adapt to a specific image, so scenes with strong geometric anchors are more resistant.

Table J.14: Additional attack sample group 1: boundary-style cases with stable or weakly changed top-1 labels. Case

Clean image Adv. image

Clean top-1

Adv. top-1

Reading

0507 RS5M

beach (0.502)

beach (0.108)

The class remains beach, but confidence drops sharply after perturbation.

0377 NWPU

white road (0.140)

white road (0.046)

The freeway geometry is stable, so the perturbation mainly suppresses confidence.

0953 SkyScript

white roofs (0.079)

white roofs (0.074)

Repeated roof-like structures remain the dominant visual anchor.

0658 RS5M

harbor (0.209)

harbor (0.213)

Large water and harbor cues remain inside the same semantic neighborhood.

38

C. Su et al.

Table J.15: Additional attack sample group 2: confidence suppression with nearby scene semantics. Case

Clean image Adv. image

Clean top-1

Adv. top-1

Reading

0403 RS5M

beach (0.414)

beach (0.058)

The top-1 remains beach, but the clean confidence is strongly reduced.

0018 RS5M

port (0.439)

port (0.146)

Port layout survives, but the adversarial input reduces the clean confidence.

0102 RSICD

grey river (0.187)

river (0.087)

The prediction stays in the same river neighborhood with lower confidence.

0227 RS5M

island (0.631)

harbor (0.092)

Coastline evidence shifts to a nearby harbor interpretation.

Table J.16: Additional attack sample group 3: clear semantic flips. Case

Clean image Adv. image

Clean top-1

Adv. top-1

Reading

0538 RSICD

stadium (0.200)

parking lots (0.060)

Sports-field evidence is replaced by parking-lot texture.

0516 RSICD

rectangular farmland (0.090)

tall buildings (0.186)

Smooth field-like evidence turns into building-like response.

0700 RS5M

beach (0.253)

tall buildings (0.562)

Shoreline evidence is overtaken by high-response structural texture.

0412 RSICD

white roofs (0.061)

church (0.207)

Roof-like evidence is redirected to an object-like class.

Thermal-Airflow Attack against IR Remote Sensing VLMs

39

Table J.17: Additional attack sample group 4: source-diverse semantic flips. Case

Clean image Adv. image

Clean top-1

Adv. top-1

Reading

0447 NWPU

airplane (0.036)

church (0.242)

Sparse desert cues are replaced by compact bright structures.

0990 SkyScript

rectangular farmland (0.037)

tall buildings (0.495)

Field-like response flips to high-confidence building semantics.

0271 SkyScript

square (0.128)

church (0.379)

Compact geometry is reinterpreted as an object-like landmark.

0801 NWPU

sparse residential (0.035)

tall buildings (0.509)

Residential evidence is amplified into tall-building semantics.

Table J.18: Representative boundary cases. These classes are not failure modes of the implementation; rather, they identify scene types where the universal perturbation has less room to override strong geometric evidence.

Class or scene type n ASR (%) Dense residential 19

10.5

Storage tanks

8

12.5

Parking lot

8

25.0

Forest

19

31.6

Tennis court

8

37.5

Interpretation Repeated roofs and block layouts provide strong geometric anchors, so the airflow texture often changes confidence without flipping the top-1 class. Circular tank geometry is visually distinctive and remains stable under the smooth perturbation. Vehicle grids and regular lane structure remain salient in many examples. Texture changes can alter confidence, but the global low-contrast vegetation pattern often remains inside the same semantic neighborhood. Court-line geometry is compact and high-contrast, making it harder for a universal thermal field to erase completely.

40

K

C. Su et al.

Full VLM Prompts and Scoring Protocol

The downstream VLM evaluation uses the same four prompts for all six models and all attack methods. Captioning is allowed 128 new tokens, while the three question-answering tasks are allowed 64 new tokens. Generation is deterministic with greedy decoding (do_sample=False, num_beams=1) and each model’s native preprocessing and adapter defaults. The prompts are deliberately short and task-specific so that changes under attack reflect image-conditioned model behavior rather than prompt engineering. Table K.19 lists the exact prompts. For captioning, we report ROUGE-L against the IR-aware reference caption. The scoring script also records BLEU, METEOR, and a lightweight CIDEr-style score, but ROUGE-L is used in the paper because it is stable across the six evaluated VLMs and directly comparable across methods. For scene recognition, the generated answer is normalized and matched to the annotated scene label. For object recognition, the answer is parsed as a set of object or land-cover terms and compared with the reference object set using F1. For IR-cue evaluation, the answer is checked against the expected infrared visual cues rather than against a single class label. The IR-cue prompt is an evaluation prompt designed to elicit thermal-cue descriptions; the metric should therefore be read as cue-fidelity under a fixed rubric, not as an unconstrained test of whether a model independently discovers thermal evidence. We report all clean and adversarial scores on the same 1,000 images and use the statistical protocol described in Appendix D.4. Table K.19: VLM prompts and reported metrics. Lower ROUGE-L, scene accuracy, and object F1 indicate stronger degradation; IR-cue accuracy is interpreted separately because attacked models may hallucinate plausible thermal cues. Task

Prompt

Max tokens

Metric

Caption

Describe this infrared remote sensing image in detail. What is the main scene type in this infrared remote sensing image? Answer with a concise scene label only. What are the main objects or land-cover elements visible in this infrared remote sensing image? Answer with a concise comma-separated list. What infrared-specific visual cues can be observed in this image? Mention visible grayscale intensity, high contrast, bright structures, low-texture or dark regions, and structural outlines if present.

128

ROUGE-L

64

Scene accuracy

64

Object F1

64

IR-cue accuracy

Scene Objects

IR cues

Thermal-Airflow Attack against IR Remote Sensing VLMs

L

41

Per-Source and Per-Class Diagnostic Results

The main paper reports aggregate ASR across models. To test whether the attack is driven by a single source dataset, we also compute source-wise ASR on the 1,000-image diagnostic split. This analysis is secondary because the diagnostic split is smaller than the full test set, but it is useful for identifying systematic source effects. Table L.20: Source-wise ASR (%) on the 1,000-image diagnostic split. The same images are used for all five CLIP backbones. Source

n OpenAI-L14 OpenAI-B32 OpenCLIP-B32 RemoteCLIP-B32 GeoRSCLIP-B32 Mean

NWPU 205 RS5M 332 RSICD 166 RSITMD 83 SkyScript 214

45.4 41.3 47.0 48.2 63.6

60.0 55.4 53.6 57.8 71.0

46.3 47.0 50.0 59.0 62.1

53.7 54.8 47.6 50.6 65.0

53.7 45.5 56.6 59.0 72.9

51.8 48.8 51.0 54.9 66.9

The attack transfers across all five sources, with SkyScript showing the largest mean ASR and RS5M the smallest. Because the perturbation is universal and not source-conditioned, this gap most likely reflects differences in image composition and caption-derived labels rather than a dataset-specific fitting effect. The source-wise trend is consistent across backbones: all models are substantially affected on every source, and no source collapses to near-zero attack success. We further inspect class-level behavior on the OpenAI-CLIP-B32 diagnostic classifier output, using only classes with at least five samples after merging minor spelling variants such as underscore-separated and space-separated labels. Table L.22 reports representative high- and low-ASR classes. Classes dominated by broad thermal texture or weak local structure, such as desert, cloud, beach, playground, and water, are especially vulnerable. Classes with repeated highcontrast man-made structure, such as dense residential areas, storage tanks, parking lots, and tennis courts, are more resistant. This supports the interpretation that AirflowAttack mainly perturbs global thermal texture and scene-level evidence, while compact geometric structure can still anchor the model prediction. Table L.22: Representative high- and low-ASR classes on OpenAI-CLIP-B32 in the diagnostic split (classes with ≥ 5 samples). Group Metric

High

Class

desert

cloud

beach

n ASR (%)

9 100.0

6 100.0

11 90.9

Class Low

Classes

n ASR (%)

playground 8 87.5

dense storage parking palace resid. tanks lot 19 8 5 8 10.5 12.5 20.0 25.0

freeway

school

7 85.7

6 83.3

20 80.0

medium resid. 14 35.7

tennis court 8 37.5

forest 19 31.6

stadium mountain 13 76.9 parking 28 39.3