Robustifying Vision-Language Models via Test-Time Prompt Adaptation
Xingyu Zhu 1 2 Huanshen Wu 2 Shuo Wang 2 * Beier Zhu 2 Jiannan Ge 2 Jiaheng Zhang 1 Long Chen 3
arXiv:2607.09450v1 [cs.CV] 10 Jul 2026
Abstract
(a)
Pre-trained Vision-Language Models (VLMs) such as CLIP achieve strong zero-shot generalization, but their performance degrades sharply under adversarial perturbations. Existing test-time adaptation methods typically rely on sample-level confidence heuristics, overlooking the intrinsic distributional structure of the data. This samplecentric approach limits robustness, as it fails to distinguish confident adversarial mispredictions from true semantic consistency. In this work, we observe that adversarial distortion is structurally brittle: while holistic representations are corrupted, semantic integrity is often preserved in the distribution of augmented views. Motivated by this insight, we propose RITA, a Robust testtIme prompT Adaptation framework that shifts from sample-level estimates to distribution-level alignment. Specifically, RITA employs optimal transport to align the distribution of augmented visual features with textual prototypes, mitigating adversarial outliers and rectifying cross-modal semantic misalignment. Furthermore, we introduce a dynamic cache to progressively accumulate reliable cues from the test stream for online refinement. Extensive experiments demonstrate that RITA significantly improves adversarial robustness without compromising clean accuracy.
(c)
Prompt Prompt
Alignment
Cache Updating
(b)
(d)
Figure 1. Augmented views retain more semantic cues under adversarial perturbations, enabeling a cache for distribution alignment that improves adversarial performance. (a) Visualization of adversarially perturbed images, where each point represents an image and different colors denote ground-truth classes. (b) Visualization of multiple augmented views generated from the same adversarial image, colored by class label, where semantic structure partially re-emerges with improved class separability compared to (a). (c) Our method leverages the selected augmented views as a cache and aligns them with textual prompts. (d) Performance comparison across different VLM backbones, demonstrating improved robustness under adversarial attacks.
cause severe performance degradation (Szegedy et al., 2014; Madry et al., 2018; Zhu et al., 2024a; 2025b), posing security risks in real word applications.
1. Introduction
Existing efforts to enhance the robustness of VLMs generally fall into two categories. The first line of work utilizes adversarial training (Mao et al., 2023; Schlarmann et al., 2024; Wang et al., 2024; Zhang et al., 2024), which aims to immunize models by explicitly integrating adversarial examples into the optimization loop. While effective, these approaches typically incur prohibitive computational costs due to on-the-fly attack generation and require access to taskspecific labeled data, thereby undermining the scalability and zero-shot flexibility inherent to foundation models. The second line of work explores test-time prompt tuning (Yoon et al., 2024; Shu et al., 2022; Zhao et al., 2025), an efficient paradigm that adapts learnable prompt contexts or predictions during inference without modifying model parameters. However, most existing test-time methods (Wang
Vision-Language Models (VLMs) (Li et al., 2022; Alayrac et al., 2022; Li et al., 2023; Zhu et al., 2024b; 2026c) like CLIP (Radford et al., 2021), pre-trained on massive imagetext pairs, have achieved remarkable zero-shot generalization. Despite this success, VLMs remain highly vulnerable to adversarial perturbations: imperceptible noise can 1 National University of Singapore 2 University of Science and Technology of China 3 The Hong Kong University of Science and Technology. * Correspondence to: Shuo Wang <[email protected]>.
Proceedings of the 43 rd International Conference on Machine Learning, Seoul, South Korea. PMLR 306, 2026. Copyright 2026 by the author(s).
1
Robustifying Vision-Language Models via Test-Time Prompt Adaptation
et al., 2025; Sheng et al., 2025) primarily rely on samplelevel confidence (e.g., entropy) to filter augmented views. These methods treat augmentations as isolated data points, overlooking the intrinsic distributional structure and latent semantics. Consequently, they fail to distinguish between confident adversarial mispredictions and true semantic consistency, limiting their effectiveness under attack.
• We propose RITA, a robust test-time prompt adaptation framework that leverages augmented views to rectify adversarial misalignment at the distribution level.
To address the above limitation, we revisit how adversarial perturbations interact with test-time augmentations in VLMs. A typical white-box attack is crafted on the original input image to maximally disrupt image-text matching based on its holistic representation. This can drastically shift the attacked image embedding in the feature space and make samples from different classes highly entangle as shown in Figure 1(a). Importantly, this adversarial effect is not equally persistent across augmentations. Geometric transformations such as random cropping and flipping change the spatial correspondence of pixels, so the perturbation pattern optimized for the original configuration becomes partially mismatched after transformation. Empirically, different views exhibit heterogeneous behaviors: while some views remain strongly influenced by the attack, others still produce confident and semantically consistent predictions (Figure 1(b)). This observation suggests that robustness should be built by exploiting the distribution of augmented views and the semantic relations it contains, rather than relying solely on sample-level confidence heuristics.
• We conduct comprehensive evaluations, showing that RITA consistently outperforms existing test-time adaptation methods.
• We formulate cross-modal semantic alignment as an Optimal Transport problem, complemented by a dynamic cache for progressive refinement.
2. Related Work Adversarial Defense in VLMs. The vulnerability of VLMs (e.g., CLIP (Radford et al., 2021)) to adversarial perturbations remains a critical challenge (Dong et al., 2018; Madry et al., 2018; Zhao et al., 2023; Zhu et al., 2026d). Attacks have evolved from uni-modal perturbations (Carlini & Wagner, 2017a; Xie et al., 2019) to multi-modal strategies like Co-Attack (Zhang et al., 2022a) that disrupt cross-modal alignment. To enhance robustness, prior defenses utilize training-time strategies, primarily adversarial contrastive tuning (Mao et al., 2023; Schlarmann et al., 2024; Wang et al., 2024). However, these methods typically require expensive re-training and labeled data, limiting practicality. Consequently, inference-time robustness has emerged to secure models without weight updates, such as diffusion purification (Feng et al., 2023; 2025) and optimization-based methods (Wang et al., 2021; Zhang et al., 2022b). Notably, Test-Time Prompt Tuning approaches like TAPT (Wang et al., 2025) and R-TPT (Sheng et al., 2025) adapt prompts using unlabeled data via contrastive learning or entropy minimization. Nevertheless, these paradigms typically treat visual inputs as isolated points, failing to exploit the underlying distributional geometry of adversarial samples. In contrast, we propose RITA, a test-time prompt tuning framework that shifts from point-level alignment to distributionlevel modeling.
Motivated by the above observation, we propose RITA, a robust test-time framework that shifts from sample-level matching to distribution-level alignment. Instead of relying on a single image embedding, RITA models the visual input as a discrete distribution over augmented views. To bridge these visual features with textual prototypes, we formulate the alignment objective as an Optimal Transport (OT) (Cuturi, 2013) problem. As illustrated in Figure 1(c), this formulation enables us to evaluate the global geometric correspondence between the visual distribution and textual representations, mitigating the influence of adversarial outliers to rectify semantic misalignment. Moreover, test samples arrive as a continuous stream, providing additional information beyond a single image. To leverage this property, RITA incorporates a dynamic cache mechanism that progressively accumulates reliable semantic views, and uses them to further refine distribution alignment online. As demonstrated in Figure 1(d), this progressive adaptation significantly enhances zero-shot robustness and maintaining competitive performance on clean data.
Optimal transport. Optimal Transport (OT) (Cuturi, 2013) provides a principled way to compare probability distributions by accounting for the geometry of the underlying feature space. With efficient solvers such as Sinkhorn (Altschuler et al., 2017; Mensch & Peyré, 2020), OT has been widely used in generative modeling (Arjovsky et al., 2017), structural alignment (Xu et al., 2019), and domain adaptation (Courty et al., 2016). In vision–language learning, OT has also been applied to reduce semantic misalignment, including few-shot learning (Lazarou et al., 2021), distribution calibration (Guo et al., 2022; Damodaran et al., 2018; Zhu et al., 2025a;b), and prompt learning (Chen et al., 2023; Wang et al., 2023; Ren et al., 2025). For example, PLOT (Chen et al., 2023) aligns image features with multiple prompts via OT-based matching to capture
Extensive experiments on multiple standard benchmarks under diverse adversarial attacks demonstrate that RITA significantly enhances zero-shot robustness while preserving competitive performance on clean data. Our contributions are summarized as follows:
2
Robustifying Vision-Language Models via Test-Time Prompt Adaptation
Optimal transport. Optimal Transport (OT) (Cuturi, 2013) provides a principled way to measure the discrepancy between two probability distributions. Consider PN two discrete distributions in the feature space, P = n=1 an δxn and P m m Q= M m=1 b δz , where δv denotes the Dirac delta function at location v, and a ∈ ∆N , b ∈ ∆M are probability vectors. Given a cost matrix C ∈ RN ×M , where Cnm measures the transport cost from xn to zm , the entropyregularized OT distance is defined as:
diverse semantics, while ALIGN (Wang et al., 2023) further introduces hierarchical/token-level transportation for fine-grained cross-modal alignment (Zhu et al., 2026a;b). AWT (Zhu et al., 2024c) similarly formulates image-text distance as an OT problem to model semantic correlations in the joint space. However, these approaches primarily focus on representation enhancement in clean settings. They rely on undistorted visual manifolds and do not account for the severe structural perturbations caused by adversarial attacks. Differing from these approaches, RITA repurposes OT for adversarial defense, aiming to reconstruct the distributionlevel correspondence disrupted by attacks. This enables inference-time correction of structural misalignment without parameter updates, ensuring robust deployment.
dOT (P, Q; C) =
(3)
×M where Π(a, b) = {T ∈ RN | T1MP= a, T⊤ 1N = b} + is the transport polytope, h(T) = − n,m Tnm log Tnm is the entropic regularization term, and λ ≥ 0 controls the regularization strength. This formulation enables efficient computation via the Sinkhorn algorithm.
3. Method An overview of RITA is illustrated in Figure 2. We first introduce the preliminaries in Sec. 3.1, then detail our distributed feature modeling and dynamic distribution alignment in Sec. 3.2 and 3.3, respectively. Finally, we provide a theoretical justification in Sec. 3.4.
3.2. Distributed Features Modeling In this section, we present our core strategy for robust testtime inference. Moving beyond vulnerable holistic image embeddings, we model the adversarial image and textual prompts as discrete distributions and align them structurally using optimal transport.
3.1. Preliminary Test-time prompt tuning. Test-Time Prompt Tuning (TPT) (Shu et al., 2022) improves the zero-shot generalization of CLIP (Radford et al., 2021) by adapting textual prompts at inference time, without accessing labeled data or updating CLIP parameters. Given a test image xt , TPT constructs a set of N stochastic augmentations Xt = {xnt }N n=1 . For each augmented view xnt , the visual representation is xnt = Φimg (xnt ), where Φimg (·) denotes the CLIP image encoder. On the text side, the prompt for class k is formulated as zk = {ω1 , ω2 , . . . , ωL , ck }, where ck is the token embedding of the class name, and ω = {ωℓ }L ℓ=1 are learnable context vectors shared across classes. The textual representation is zk = Φtext (zk ), with Φtext (·) being the CLIP text encoder. For each view xnt , the prediction probability is computed by feature matching: exp cos(xnt , zk )/τ p(k | xnt ; ω) = PK (1) , n j=1 exp cos(xt , zj )/τ
Adversarial perturbations. Adversarial attacks aim to degrade model predictions by introducing small, carefully crafted perturbations. In a white-box setting, given a clean image x and its ground-truth label y, the adversarial example x′ is generated by maximizing the cross-entropy loss within an ℓp -norm constraint: x′ =
argmax
LCE p(· | x′ ; z), y ,
(4)
∥x′ −x∥p ≤ϵadv
where ϵadv denotes the perturbation budget, and p(· | x′ ; z) represents the probability distribution over K classes as defined in Eq. (1). Typically, we approximate this optimization using the iterative Projected Gradient Descent (PGD) (Madry et al., 2018). Multi-prototype distribution alignment. While adversarial attacks distort alignment at the global representation level, relying on single-point embeddings is insufficient to capture the semantic variations. To alleviate this, we construct sets of diverse visual and textual representations and model their correspondence at the distribution level.
where τ is the temperature parameter and K is the number of categories. Specifically, TPT employs a confidence selection strategy to filter out unreliable augmentations. It selects a subset S of low-entropy views to Pcompute the naggregated 1 prediction p̄(k | xt ; ω) = |S| n∈S p(k | xt ; ω). The prompt ω is then optimized by minimizing the entropy of this aggregated distribution p̄:
Specifically, given an adversarial image x̂t , we apply data augmentations to obtain N views {x̂nt }N n=1 , producing visual features {x̂nt }N n=1 . Instead of a single text prototype, (m) for each class k, we construct M prompts {zk }M m=1 using learnable context vectors, yielding textual features M {zm k }m=1 . We model the adversarial image and the k-th
K X ω ∗ = argmin − p̄(k | xt ; ω) log p̄(k | xt ; ω) . (2) ω
min ⟨T, C⟩ − λh(T), T∈Π(a,b)
k=1
3
Robustifying Vision-Language Models via Test-Time Prompt Adaptation
...
Class Name Class Name
...
Textual Encoder
...
...
Class Name
...
Optimal Transport Augment
...
...
Visual Encoder
Cache
Updating
class k:
...
class k+1:
...
...
Optimal Transport Gradient
Modality Alignment
...
Figure 2. Overview of the proposed RITA framework. Given an adversarial test image, RITA extracts multi-view visual features and class-specific textual prototypes using a frozen CLIP encoder. Both modalities are modeled as discrete distributions and aligned via entropy-regularized optimal transport. Low-entropy views are used to update a dynamic cache of reliable semantics.
Confidence-based cache update. To update the cache with reliable samples, we evaluate the confidence of each individual augmented view. Instead of solving the full global transport problem, we quantify the instance-level alignment by measuring the average affinity between the view x̂nt and the text distribution of class k: P τM 1 ) exp M cos(x̂tn , zm k m=1 P . (8) pnt (k) = P τM K 1 n m m=1 cos(x̂t , zj ) j=1 exp M
class prototype as discrete distributions:
Pt =
N X 1 n=1
N
δx̂nt ,
Qk =
M X 1 δzm . k M m=1
(5)
Here, we assign uniform weights to both modalities, i.e., 1 at = N1 1N and bk = M 1M , such that the marginal constraints of OT are satisfied. We then measure the distributionlevel alignment between the visual distribution Pt and the textual prototype distribution Qk via entropy-regularized optimal transport:
PK We compute entropy H(pnt ) = − k=1 pnt (k) log pnt (k) and retain confident views: Bt = (x̂tn , ŷtn ) H(pnt ) ≤ γ, ŷtn = argmax pnt (k) .
dOT (Pt , Qk ; Ct,k ) = min (⟨Tt,k , Ct,k ⟩ − λh(Tt,k )) , Tt,k ∈Π
(6) ×M where Tt,k ∈ RN is the transport plan, and the cost + matrix is defined by the cosine distance, Ct,k (n, m) = 1 − cos(x̂tn , zkm ). Intuitively, a smaller OT distance indicates better alignment between the test image and class k at the distribution level. Accordingly, we predict the label by selecting the class with the minimum transport cost: ŷ = argmin dOT (Pt , Qk ; Ct,k ).
k
(9) We maintain a class-wise cache {X̂k }K k=1 , where X̂k ∈ RNk ×d stores cached visual features pseudo-labeled as class k (Nk is the current cache size). The cache is updated online by prioritizing lower-entropy samples. To bridge the modality gap, we align cached visual features to the textual space of class k by solving an Orthogonal Procrustes (Ouali et al., 2023) problem:
(7)
k∈[K]
3.3. Dynamic Distribution Alignment
Wk∗ = argmin X̂k W − 1Nk z̄⊤ k W⊤ W=I
While distribution-level alignment in Sec. 3.2 effectively processes individual samples, it treats inference steps in isolation, neglecting the semantic consensus in the continuous test stream. To exploit this temporal information, we introduce a dynamic cache mechanism that accumulates reliable visual features to iteratively refine the alignment online.
2
, F
(10)
PM 1 m where z̄k = M m=1 zk is the mean text embedding for class k. The aligned cached features are X̃k = X̂k Wk∗ . Cache-based distribution matching. For each class k, we instantiate a discrete probability distribution over the 4
Robustifying Vision-Language Models via Test-Time Prompt Adaptation
aligned cached features:
Q̃k =
et al., 2004)), texture recognition (DTD (Cimpoi et al., 2014)), satellite imagery (EuroSAT (Helber et al., 2019)), human action recognition (UCF101 (Soomro et al., 2012)), as well as several fine-grained classification tasks, namely Pets (Parkhi et al., 2012), Cars (Krause et al., 2013), Flowers (Nilsback & Zisserman, 2008), and Aircraft (Maji et al., 2013). To further assess robustness under distribution shifts, we conduct additional evaluations on ImageNet (Deng et al., 2009) and four of its variants that share the same label space: ImageNetV2 (Recht et al., 2019), ImageNet-Sketch (Wang et al., 2019), ImageNet-A (Hendrycks et al., 2021b), and ImageNet-R (Hendrycks et al., 2021a). These benchmarks introduce significant variations in image sources, styles, and underlying visual statistics. Detailed analysis of these datasets is provided in Appendix C.1.
N
k 1 X δ j, Nk j=1 x̃k
where x̃kj ∈ X̃k .
(11)
We quantify the discrepancy via the same entropyregularized OT distance as in Sec. 3.2, but with a cachespecific cost matrix C̃t,k (n, j) = 1 − cos(x̂tn , x̃kj ), and denote it as dcOT (Pt , Q̃k ; C̃t,k ). Final inference. We classify the test sample by identifying the category that minimizes the joint transport cost, which integrates both the global prompt alignment and the local cache consensus: ŷ = argmin dOT (Pt , Qk ; Ct,k )+α dcOT (Pt , Q̃k ; C̃t,k ) , k∈[K]
Implementation details. We build all experiments on the official pre-trained CLIP models with two backbones, CLIPViT-B/32 and CLIP-ViT-B/16. We generate adversarial examples using PGD (Madry et al., 2018) under an L∞ constraint. We use ϵ = 4.0 with 7 steps for both backbones. At test time, we update only the prompt parameters while keeping the CLIP backbone freezed. The prompt number M is set to 4 and initialized with the template “a photo of a”. We use AdamW and fix the Test-Time Adaptation (TTA) step at 1 per test sample, with a learning rate of 0.005. We apply standard test-time augmentations for images, including random cropping, resizing, and horizontal flipping. For text, we use an Large Language Model (LLM) to generate class-specific descriptions(Zhu et al., 2024c). For each test image, we sample N = 64 augmented views (including the original). For cache construction, We set the entropy threshold to γ = 0.8. More experiment details are provided in Appendix B.
(12) where α ≥ 0 controls the contribution of the dynamic cache. 3.4. Theoretical Analysis Standard methods like TPT (Shu et al., 2022) and RTPT (Sheng et al., 2025) optimize centroid alignment via mean pooling and cosine similarity. For ℓ2 -normalized CLIP features, this is equivalent to minimizing the squared Euclidean distance between centroids (∥x − z∥2 = 2(1 − cos(x, z))). To reveal RITA’s geometric advantage, we analyze alignment using the 2-Wasserstein distance (W2 ). Let Pt (µx , Σx ) and Qk (µz , Σz ) be the visual and textual distributions. We obtain the following decomposition: Theorem 3.1 (Decomposition of Alignment Objective). The Optimal Transport objective (LOT ) imposes a stricter bound by decomposing into a centroid alignment term and a structural variance penalty:
Equation (13) holds exactly for Gaussian distributions and serves as a general lower bound, as derived in Appendix A. This inequality highlights a critical robustness gap: minimizing only Lmean leaves structural variance Rvar unconstrained, allowing attackers to distort distribution geometry.
Comparison methods. We compare RITA with CLIPbased test-time adaptation baselines, including TPT (Shu et al., 2022), R-TPT (Sheng et al., 2025), C-TPT (Yoon et al., 2024), and MTA (Zanella & Ayed, 2024), as well as the zero-shot CLIP baseline. We also report an Ensemble baseline that averages predictions over augmented views. To further evaluate compatibility with robust pre-trained weights, We incorporate three representative adversarially fine-tuned CLIP models, namely TeCoA (Mao et al., 2023), PMG (Wang et al., 2024), and FARE (Schlarmann et al., 2024). All methods follow the instance-level test-time adaptation protocol: each test sample is adapted and predicted independently, without access to other test samples.
4. Experiments
4.2. Main Results
4.1. Setup
Results on fine-grained datasets. We evaluate our method on eight fine-grained benchmark datasets with ViT-B/32 and ViT-B/16 backbones, as presented in Table 1. The results show that RITA achieves the highest adversarial accuracy
W22 (Pt , Qk ) ≈ | {z } LOT (RITA)
∥µ − µ ∥2 | x {z z }
Lmean (TPT-equivalent)
+
B2 (Σx , Σz ) | {z }
,
Rvar (Structural Penalty)
(13) where B2 (A, B) = Tr(A+B−2(A1/2 BA1/2 )1/2 ) is the Bures metric, quantifying the geometric mismatch between covariances.
Datasets. We evaluate our method on eight image classification benchmarks spanning a wide range of visual domains, including generic object recognition (Caltech101 (Fei-Fei 5
Robustifying Vision-Language Models via Test-Time Prompt Adaptation Table 1. Results (%) of adaptation methods on fine-grained classification datasets with ϵ set to 1.0. Bold and underlined entries indicate the best and second-best results, respectively. Acc. denotes accuracy on clean data, and Rob. denotes accuracy under adversarial perturbations.
ViT-B/32
Caltech101 Pets Cars Flower102 Aircraft DTD EuroSAT UCF101 Avg. Acc. Rob. Acc. Rob. Acc. Rob. Acc. Rob. Acc. Rob. Acc. Rob. Acc. Rob. Acc. Rob. Acc. Rob.
CLIP 90.9 Ensemble 91.6 TPT 91.4 C-TPT 91.8 MTA 91.8 R-TPT 90.6 RITA 92.3
25.6 85.4 77.6 75.9 80.8 86.2 86.5
83.0 2.1 85.0 73.4 84.1 58.6 84.9 52.2 85.8 62.6 84.5 73.1 85.9 74.5
49.7 0.0 57.8 39.6 62.9 30.8 60.8 27.1 64.1 34.5 63.1 44.6 59.6 42.9
65.8 2.2 67.4 57.5 63.8 43.6 65.9 42.1 64.8 44.8 62.6 53.1 68.7 58.7
18.3 0.0 20.1 14.4 19.0 10.1 17.7 8.7 20.4 11.1 19.1 12.9 20.2 15.2
40.8 5.4 46.1 39.3 42.2 28.6 44.3 27.1 44.0 29.3 42.1 36.7 46.2 40.1
18.6 0.1 32.5 23.6 35.1 15.1 34.7 9.0 34.5 7.8 32.0 22.4 33.4 24.8
62.1 2.4 61.6 53.5 62.3 38.1 62.6 35.3 63.6 40.1 62.8 54.2 62.8 55.0
53.6 4.7 57.8 48.3 57.6 37.8 57.8 34.6 58.6 38.8 57.1 47.9 58.6 49.7
ViT-B/16
Method
CLIP 85.9 Ensemble 92.1 TPT 94.1 C-TPT 93.9 MTA 94.3 R-TPT 93.7 RITA 93.8
10.8 87.4 79.6 76.5 81.9 87.8 88.5
83.5 0.5 88.7 77.2 87.4 62.8 88.2 55.8 88.0 64.5 87.2 74.7 89.8 77.3
55.7 0.0 63.2 46.7 66.5 35.5 65.8 30.5 67.7 38.2 67.0 46.9 64.2 47.1
61.7 0.1 70.8 59.9 66.1 48.3 69.6 45.5 65.0 46.9 68.7 55.7 71.6 61.3
15.7 0.0 25.9 17.9 23.4 12.3 23.9 9.8 24.0 12.6 23.9 17.3 26.2 19.2
40.4 2.4 50.9 43.2 45.9 29.1 45.9 26.6 46.5 28.7 46.4 39.7 51.5 44.7
23.7 0.0 32.9 26.7 42.6 7.4 42.3 7.1 42.5 13.7 34.7 26.8 33.4 27.6
58.9 0.5 64.6 54.3 67.9 39.7 65.6 34.7 67.5 40.8 67.2 55.4 65.5 55.8
53.2 1.8 61.1 51.6 61.7 39.3 61.9 35.8 61.9 40.9 61.1 50.5 62.0 52.7
Table 2. Classification accuracy (%) on 8 datasets using different adversarially finetuned CLIP models. Method
Caltech101 Pets Cars Flower102 Aircraft DTD EuroSAT UCF101 Avg. Acc. Rob. Acc. Rob. Acc. Rob. Acc. Rob. Acc. Rob. Acc. Rob. Acc. Rob. Acc. Rob. Acc. Rob.
TeCoA + Ensemble + MTA + R-TPT + RITA
77.6 76.0 79.5 77.1 78.1
64.3 68.3 56.5 70.3 70.4
59.8 61.3 61.6 60.9 62.3
39.9 54.8 28.8 54.0 54.9
20.6 19.4 20.9 23.1 21.1
9.3 14.5 10.4 17.8 15.7
37.1 38.4 37.2 35.4 39.2
23.9 33.2 24.0 30.8 34.0
5.7 6.9 6.6 7.0 7.1
2.8 4.9 3.8 4.7 5.5
24.1 27.5 25.8 26.7 27.8
14.4 25.2 19.7 24.7 25.5
15.9 11.5 12.2 12.0 12.2
12.4 12.0 11.2 11.8 12.1
40.5 39.4 42.0 40.1 40.9
23.4 34.5 17.3 35.0 35.8
35.2 35.1 35.7 35.3 36.1
23.8 30.9 21.4 31.1 31.7
PMG + Ensemble + MTA + R-TPT + RITA
82.3 78.9 79.5 79.3 79.9
70.8 71.7 65.4 73.2 73.7
61.8 60.7 61.8 62.0 62.5
41.4 54.1 31.3 55.1 55.4
24.7 16.6 17.9 18.3 18.9
12.5 11.6 12.9 14.9 13.9
36.2 37.8 36.7 35.3 38.7
25.3 32.7 21.6 30.4 33.9
5.2 7.2 5.8 5.4 7.5
2.9 5.1 3.3 4.1 5.4
22.8 26.6 22.9 25.4 27.4
16.0 25.2 18.7 23.0 25.4
17.1 14.0 13.8 13.2 14.4
12.8 13.9 12.5 12.9 14.1
42.6 42.0 43.1 42.3 43.5
27.8 37.7 22.6 38.2 38.6
36.5 35.5 35.1 35.2 36.6
26.1 31.5 23.5 31.4 32.6
FARE + Ensemble + MTA + R-TPT + RITA
86.6 86.1 87.7 86.5 86.8
62.9 80.2 70.0 81.1 81.9
77.7 77.9 78.4 77.4 78.7
38.1 70.1 45.0 70.1 70.5
40.4 38.8 40.6 43.0 41.2
9.7 29.5 24.8 33.0 31.2
48.7 48.9 49.2 46.2 50.1
22.5 42.3 30.5 40.2 43.1
10.2 10.5 11.0 10.0 11.8
2.3 7.8 6.6 7.4 8.6
32.4 36.9 32.8 33.6 37.5
18.1 32.4 25.3 30.0 32.9
22.4 13.5 13.5 12.9 14.2
11.0 11.6 11.8 12.1 12.7
52.9 52.8 53.9 53.8 54.3
22.2 45.6 29.8 46.8 47.0
46.4 45.6 45.8 45.4 46.8
23.3 39.9 30.4 40.0 41.0
Results with adversarially finetuned CLIP models. As an inherently plug-and-play framework, we integrated RITA with three representative adversarially fine-tuned models, with results summarized in Table 2. RITA exhibits significant synergy with these robust baselines, substantially enhancing robust accuracy across all datasets without sacrificing clean performance. Notably, when combined with FARE, RITA achieves a striking 81.9% robust accuracy on Caltech101, representing a 19.0% improvement over the baseline. These results demonstrate that RITA is a versatile adaptation framework that can seamlessly fortify existing adversarial defense models during inference.
on nearly all datasets. Compared to vanilla CLIP, RITA improves average robustness by 45.0% and 50.9% with ViTB/32 and ViT-B/16, respectively. This effectively mitigates the severe vulnerability of the baseline under adversarial attacks, which can cause near-zero accuracy on datasets such as Cars and Aircraft. Furthermore, compared to R-TPT, the state-of-the-art test-time adaptation method, RITA achieves consistent gains in average robust accuracy, outperforming it by 1.8% and 2.2% on the two architectures. Meanwhile, our method also achieves the highest average accuracy on clean samples.This demonstrates that RITA enhances adversarial robustness while preserving recognition performance in attack-free environments with negligible degradation. Notably, RITA performs better with ViT-B/16 than with ViTB/32, aligning with the intuition that a fine-grained ViT backbone yields stronger recognition capabilities.
Results under different attack types. Table 3 further assesses the generalizability of RITA against CW (Carlini & Wagner, 2017b) and DI (Xie et al., 2019) attacks. RITA consistently demonstrates superior defense across protocols,
6
Robustifying Vision-Language Models via Test-Time Prompt Adaptation
Caltech101 Pets Cars Flower102 Aircraft DTD EuroSAT UCF101 Avg. Acc. Rob. Acc. Rob. Acc. Rob. Acc. Rob. Acc. Rob. Acc. Rob. Acc. Rob. Acc. Rob. Acc. Rob.
CW
CLIP Ensemble TPT C-TPT MTA R-TPT RITA
85.9 92.1 94.1 93.9 94.3 93.7 93.8
22.7 86.7 77.2 76.0 79.2 88.1 87.8
83.5 88.7 87.4 88.2 88.0 87.2 89.8
7.7 78.6 66.9 62.1 67.8 74.4 78.7
55.7 63.2 66.5 65.8 67.7 67.0 64.2
6.6 50.5 43.2 39.6 43.4 50.7 52.6
61.7 70.8 66.1 69.6 65.0 68.7 71.6
5.8 61.6 49.9 47.5 48.4 55.7 62.0
15.7 25.9 23.4 23.9 24.0 23.9 26.2
8.8 22.2 16.1 15.1 16.1 20.1 22.9
40.4 50.9 45.9 45.9 46.5 46.4 51.5
16.3 45.2 30.7 30.2 30.6 39.6 46.2
23.7 32.9 42.6 42.3 42.5 34.7 33.4
15.4 24.8 13.5 11.5 19.3 24.8 25.2
58.9 64.6 67.9 65.6 67.5 67.2 65.5
11.3 55.3 44.1 40.3 44.6 56.2 56.8
53.2 61.1 61.7 61.9 61.9 61.1 62.0
11.8 53.1 42.7 40.2 43.6 51.2 54.0
DI
CLIP Ensemble TPT C-TPT MTA R-TPT RITA
85.9 92.1 94.1 93.9 94.3 93.7 93.8
23.0 84.3 80.7 79.5 82.6 84.9 84.7
83.5 88.7 87.4 88.2 88.0 87.2 89.8
4.4 69.1 65.2 59.5 65.6 66.7 69.4
55.7 63.2 66.5 65.8 67.7 67.0 64.2
0.6 39.1 38.3 34.2 39.5 39.1 39.9
61.7 70.8 66.1 69.6 65.0 68.7 71.6
1.9 52.7 49.7 47.3 48.0 48.1 53.4
15.7 25.9 23.4 23.9 24.0 23.9 26.2
0.0 15.3 13.5 11.6 13.5 14.1 15.9
40.4 50.9 45.9 45.9 46.5 46.4 51.5
6.1 39.7 30.3 29.1 30.9 35.6 39.8
23.7 32.9 42.6 42.3 42.5 34.7 33.4
0.0 17.6 7.4 7.4 14.7 18.0 18.4
58.9 64.6 67.9 65.6 67.5 67.2 65.5
3.2 47.6 40.5 37.0 41.9 47.1 47.7
53.2 61.1 61.7 61.9 61.9 61.1 62.0
4.9 45.7 40.7 38.2 42.1 44.2 46.2
Clean Robust
4 8 16 32 Number of Learnable Prompts (a) Prompt Number
52 50 48 46 44
Clean Robust
32 64 96 128 Number of Augmented Views
53 51 49 47 45 43
70 Clean Robust
0.0 0.2 0.4 0.6 0.8 1.0 Cache Weight (a) DTD
(b) Augmentation View
Accuracy (%)
50 40 30 20 10 0
Accuracy (%)
Method
Accuracy (%)
Accuracy (%)
Table 3. Results (%) of adaptation methods on fine-grained classification datasets under different attacks using ViT-B/16 with ϵ = 1.0.
66 Clean Robust
62 58 54
0.0 0.2 0.4 0.6 0.8 1.0 Cache Weight (b) ImageNet
Figure 4. Sensitivity analysis of the cache integration coefficient α. Classification accuracy (%) on (a) DTD and (b) ImageNet is evaluated across α values using ViT-B/16 with ϵ = 1.0.
Figure 3. Ablation studies of key hyperparameters in DTD dataset using ViT-B/16 with ϵ = 1.0. (a) Performance variation with respect to the number of learnable prompts. (b) Impact of the number of augmentation views during inference.
Contribution coefficient of dynamic cache. We investigate the sensitivity of RITA to the integration coefficient α during inference on the DTD and ImageNet datasets, as shown in Figure 4. Experimental results confirm that the cache mechanism effectively boosts performance. On both datasets, increasing α from 0 to 0.1 yields a significant accuracy gain, followed by a slight upward trend that gradually stabilizes as α further scales. These results indicate that the visual priors stored in the cache not only play a crucial role in correcting predictions under adversarial settings, but also offer effective semantic complementarity during the inference stage. Ablation study of the dynamic cache is provided in Appendix D.1.
achieving a leading average accuracy of 54.0% under CW attacks. Under the more challenging DI attack, RITA maintains its advantage with 46.2% accuracy, surpassing R-TPT by 2.0%. These results suggest that RITA enhances the intrinsic robustness of VLMs against diverse adversarial threats via its effective test-time adaptation mechanism. 4.3. Ablation Study Number of learnable prompts and augmented views. As illustrated in Figure 3, we conduct sensitivity analyses on the DTD dataset regarding the number of learnable prompts and augmentation views. Figure 3(a) shows that increasing the number of learnable prompts from 4 to 32 leads to steady performance gains across both settings. Figure 3(b) assesses the sensitivity of augmentation view scales during inference, where adversarial accuracy peaks at 96 views. Notably, scaling to 128 views results in a slight performance decline, a trend consistently observed in noise-free environments as well. Consequently, selecting a moderate number of augmentation views enables a superior trade-off between efficiency and accuracy.
Different perturbation budgets. To assess the resilience of RITA under various attack intensities, we conduct ablation studies on all fine-grained datasets and ImageNet dataset, with the results illustrated in Figure 5. As the perturbation budget ϵ increases from 1 to 4, the accuracy declines as expected, indicating the generation of more potent perturbations. Simultaneously, we observe that increasing the number of TTA steps generally yields additional and consistent defensive gains across evaluations. 7
50
52.7 +0.5+0.2
45.1 +0.6+0.3
35.0 +0.5+0.3
25 0
75 50
55.1 +0.5+0.1
34.1 +0.4+0.2
25 0
1.0 2.0 4.0 Attack Strengths
45.8 +0.6+0.3
Table 4. Comparison of running time on fine-grained datasets using ViT-B/16 with ϵ = 1.0.
TTA step = 1 TTA step = 2 TTA step = 4
Model TPT C-TPT MTA R-TPT RITA
1.0 2.0 4.0 Attack Strengths
(a) Fine-grained datasets
(b) ImageNet
50 40 30 20 10 0
RN50 ViT-B/16 ViT-B/32
0
1 2 4 Number of TTA Steps
(a) Fine-grained datasets
Accuracy (%)
Accuracy (%)
50 40 30 20 10 0
Density
Figure 5. Adversarial robustness (%) under varying perturbation budgets and TTA steps on (a) fine-grained datasets and (b) ImageNet. Robust accuracy is evaluated using ViT-B/16 under ϵ ∈ {1.0, 2.0, 4.0} and TTA steps ∈ {1, 2, 4}. +x.x indicates the increment relative to the case where TTA step = 1.
1.52s/image 1.64s/image 1.20s/image 1.70s/image 1.76s/image
6
original attacked views cache augmented views
3.6
3.7 3.8 3.9 KL Divergence
4
Clean
Robust
61.7 61.9 61.9 61.1 62.0
39.3 35.8 40.9 50.5 52.7
original attacked views cache augmented views
2 0
(a) DTD
RN50 ViT-B/16 ViT-B/32
0
6 4 2 0
Accuracy
Running Time
Density
TTA step = 1 TTA step = 2 TTA step = 4
75
Accuracy (%)
Accuracy (%)
Robustifying Vision-Language Models via Test-Time Prompt Adaptation
4.2 4.3 4.4 4.5 4.6 4.7 KL Divergence (b) Caltech101
Figure 7. Comparison of KL divergence distributions for semantic alignment between original visual features and augmented features from the cache mechanism. Kernel Density Estimation (KDE) curves are presented for (a) DTD and (b) Caltech101 datasets. Lower KL values signify more deterministic vision-text alignment.
1 2 4 Number of TTA Steps (b) ImageNet
Figure 6. Evolution of adversarial robustness with respect to TTA steps across model architectures, where (a) presents the average robust accuracy over fine-grained datasets and (b) displays results on ImageNet. TTA step = 0 denotes the CLIP baseline.
Kullback-Leibler (KL) divergence. Specifically, we measure the divergence between the class-conditional distributions over text prototypes and the ideal one-hot targets. On both DTD and Caltech101 datasets, the curves for augmented features exhibit a pronounced leftward shift and higher concentration in the low-value region compared to the original features. This demonstrates that our cache mechanism effectively rectifies adversarial semantic biases, establishing more deterministic vision-text associations. Extended analysis is provided in Appendix D.2.
Number of TTA steps. We report the average robust accuracy across three different backbones on all fine-grained datasets as shown in Figure 6(a) and the ImageNet dataset as illustrated in Figure 6(b). The experimental results indicate that all models reach their performance peak at step 2, while the accuracy slightly declines when the steps are increased to 4 across all evaluated datasets. This observation justifies our choice of a small number of iterations, e.g., TTA step = 1, which simultaneously ensures superior robustness and high computational efficiency for real-time inference.
Conclusion & Limitations In this work, we propose RITA, a robust test-time prompt adaptation framework that enhances the adversarial robustness of VLMs without requiring retraining or access to labeled data. By modeling augmented visual features and prompt-induced textual prototypes as distributions and aligning them via optimal transport, RITA corrects cross-modal semantic misalignment caused by adversarial perturbations. Furthermore, a dynamic cache mechanism progressively aggregates reliable semantic cues from the test stream to refine alignment online. Extensive experiments across diverse benchmarks, attack types, and model backbones demonstrate that RITA consistently improves adversarial robustness while preserving competitive performance on clean data. Our results suggest that distribution-level alignment is a principled and effective paradigm for robust inference in large pre-trained VLMs.
Inference efficiency analysis. Table 4 presents the average comparison of running time, clean accuracy, and robust accuracy across various methods on fine-grained datasets. The empirical results indicate that RITA achieves superior classification performance while maintaining highly competitive inference efficiency compared with existing adaptation methods. Compared to MTA, which exhibits the fastest inference speed, RITA maintains a substantial lead of 11.8% in robust accuracy. These observations demonstrate that RITA strikes an excellent balance between computational cost and model robustness, delivering more resilient and precise inference results with minimal additional time latency. Semantic alignment analysis. As shown in Figure 7, we visualize semantic alignment quality via KDE curves of the 8
Robustifying Vision-Language Models via Test-Time Prompt Adaptation
While RITA demonstrates strong image classification performance, its extension to generative tasks like image captioning remains for future exploration. We believe our distribution-level alignment provides a foundation for adaptation in these broader multimodal scenarios.
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. In CVPR, 2014.
Impact Statement
Cuturi, M. Sinkhorn distances: Lightspeed computation of optimal transport. In NIPS, pp. 2292–2300, 2013.
Courty, N., Flamary, R., Tuia, D., and Rakotomamonjy, A. Optimal transport for domain adaptation. TPAMI, 2016.
This work improves the reliability and adversarial robustness of pre-trained VLMs, which is important for their deployment in safety-sensitive applications. The proposed method operates entirely at test time, without modifying model parameters or requiring additional training data, making it lightweight and easy to integrate into existing systems. Nevertheless, it does not eliminate all security risks, and adversarial attacks may continue to evolve. Future work should further investigate robust inference-time defenses and evaluate potential failure modes and misuse risks.
Damodaran, B. B., Kellenberger, B., Flamary, R., Tuia, D., and Courty, N. Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation. In ECCV (4), volume 11208 of Lecture Notes in Computer Science, pp. 467–483. Springer, 2018. Deng, J., Dong, W., Socher, R., Li, L., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. Dong, Y., Liao, F., Pang, T., Su, H., Zhu, J., Hu, X., and Li, J. Boosting adversarial attacks with momentum. In CVPR, pp. 9185–9193. IEEE Computer Society, 2018.
Acknowledgement This research is supported by the National Natural Science Foundation of China (No. 62576330) and the National Natural Science Foundation of Anhui (No.2508085MF143).
Fei-Fei, L., Fergus, R., and Perona, P. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPR Workshops, 2004.
References
Feng, C., Yu, K., Liu, Y., Khan, S., and Zuo, W. Diverse data augmentation with diffusions for effective test-time prompt tuning. In ICCV, pp. 2704–2714. IEEE, 2023.
Alayrac, J., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick, J. L., Borgeaud, S., Brock, A., Nematzadeh, A., Sharifzadeh, S., Binkowski, M., Barreira, R., Vinyals, O., Zisserman, A., and Simonyan, K. Flamingo: a visual language model for few-shot learning. In NeurIPS, 2022.
Feng, C., He, Y., Zou, J., Khan, S. H., Xiong, H., Li, Z., Zuo, W., Goh, R. S. M., and Liu, Y. Diffusion-enhanced test-time adaptation with text and image augmentation. Int. J. Comput. Vis., 133(8):5083–5098, 2025. Guo, D., Tian, L., Zhao, H., Zhou, M., and Zha, H. Adaptive distribution calibration for few-shot learning with hierarchical optimal transport. In NeurIPS, 2022.
Altschuler, J. M., Weed, J., and Rigollet, P. Near-linear time approximation algorithms for optimal transport via sinkhorn iteration. In NIPS, pp. 1964–1974, 2017.
Helber, P., Bischke, B., Dengel, A., and Borth, D. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE J. Sel. Top. Appl. Earth Obs. Remote. Sens., 12(7):2217–2226, 2019.
Arjovsky, M., Chintala, S., and Bottou, L. Wasserstein generative adversarial networks. In ICML, 2017. Carlini, N. and Wagner, D. Towards evaluating the robustness of neural networks. In IEEE Symposium on Security and Privacy, 2017a.
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., Song, D., Steinhardt, J., and Gilmer, J. The many faces of robustness: A critical analysis of out-of-distribution generalization. In ICCV, 2021a.
Carlini, N. and Wagner, D. A. Towards evaluating the robustness of neural networks. In IEEE Symposium on Security and Privacy, pp. 39–57. IEEE Computer Society, 2017b.
Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D. Natural adversarial examples. In CVPR, 2021b.
Chen, G., Yao, W., Song, X., Li, X., Rao, Y., and Zhang, K. PLOT: prompt learning with optimal transport for visionlanguage models. In ICLR. OpenReview.net, 2023.
Krause, J., Stark, M., Deng, J., and Fei-Fei, L. 3d object representations for fine-grained categorization. In ICCV Workshops, 2013. 9
Robustifying Vision-Language Models via Test-Time Prompt Adaptation
Lazarou, M., Stathaki, T., and Avrithis, Y. Iterative label cleaning for transductive and semi-supervised few-shot learning. In ICCV, 2021.
Schlarmann, C., Singh, N. D., Croce, F., and Hein, M. Robust CLIP: unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language models. In ICML. OpenReview.net, 2024.
Li, J., Li, D., Xiong, C., and Hoi, S. C. H. BLIP: bootstrapping language-image pre-training for unified visionlanguage understanding and generation. In ICML, 2022.
Sheng, L., Liang, J., Wang, Z., and He, R. R-TPT: improving adversarial robustness of vision-language models through test-time prompt tuning. In CVPR, pp. 29958– 29967. Computer Vision Foundation / IEEE, 2025.
Li, J., Li, D., Savarese, S., and Hoi, S. C. H. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, volume 202 of Proceedings of Machine Learning Research, pp. 19730–19742. PMLR, 2023.
Shu, M., Nie, W., Huang, D., Yu, Z., Goldstein, T., Anandkumar, A., and Xiao, C. Test-time prompt tuning for zero-shot generalization in vision-language models. In NeurIPS, 2022.
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. In ICLR. OpenReview.net, 2018.
Soomro, K., Zamir, A. R., and Shah, M. UCF101: A dataset of 101 human actions classes from videos in the wild. CoRR, abs/1212.0402, 2012.
Maji, S., Rahtu, E., Kannala, J., Blaschko, M. B., and Vedaldi, A. Fine-grained visual classification of aircraft. CoRR, abs/1306.5151, 2013.
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I. J., and Fergus, R. Intriguing properties of neural networks. In ICLR (Poster), 2014.
Mao, C., Geng, S., Yang, J., Wang, X., and Vondrick, C. Understanding zero-shot adversarial robustness for largescale models. In ICLR. OpenReview.net, 2023.
Wang, D., Shelhamer, E., Liu, S., Olshausen, B. A., and Darrell, T. Tent: Fully test-time adaptation by entropy minimization. In ICLR. OpenReview.net, 2021.
Mensch, A. and Peyré, G. Online sinkhorn: Optimal transport distances from sample streams. In NeurIPS, 2020.
Wang, D., Li, M., Liu, X., Xu, M., Chen, B., and Zhang, H. Tuning multi-mode token-level prompt alignment across modalities. In NeurIPS, 2023.
Nilsback, M. and Zisserman, A. Automated flower classification over a large number of classes. In ICVGIP, 2008.
Wang, H., Ge, S., Lipton, Z., and Xing, E. P. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems, pp. 10506–10518, 2019.
Ouali, Y., Bulat, A., Martı́nez, B., and Tzimiropoulos, G. Black box few-shot adaptation for vision-language models. In ICCV, pp. 15488–15500. IEEE, 2023. Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C. V. Cats and dogs. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA, June 16-21, 2012, pp. 3498–3505. IEEE Computer Society, 2012. doi: 10.1109/CVPR.2012.6248092. URL https: //doi.org/10.1109/CVPR.2012.6248092.
Wang, S., Zhang, J., Yuan, Z., and Shan, S. Pre-trained model guided fine-tuning for zero-shot adversarial robustness. In CVPR, pp. 24502–24511. IEEE, 2024. Wang, X., Chen, K., Zhang, J., Chen, J., and Ma, X. TAPT: test-time adversarial prompt tuning for robust inference in vision-language models. In CVPR, pp. 19910–19920. Computer Vision Foundation / IEEE, 2025.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision. In ICML, volume 139 of Proceedings of Machine Learning Research, pp. 8748–8763. PMLR, 2021.
Xie, C., Zhang, Z., Zhou, Y., Bai, S., Wang, J., Ren, Z., and Yuille, A. L. Improving transferability of adversarial examples with input diversity. In CVPR, pp. 2730–2739. Computer Vision Foundation / IEEE, 2019.
Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. Do imagenet classifiers generalize to imagenet? In ICML, 2019.
Xu, H., Luo, D., and Carin, L. Scalable gromov-wasserstein learning for graph partitioning and matching. NeurIPS, 2019.
Ren, H., Tang, F., Zheng, H., Zhao, H., Guo, D., and Chang, Y. Modality-consistent prompt tuning with optimal transport. IEEE Trans. Circuits Syst. Video Technol., 35(3): 2499–2512, 2025.
Yoon, H. S., Yoon, E., Tee, J. T. J., Hasegawa-Johnson, M. A., Li, Y., and Yoo, C. D. C-TPT: calibrated test-time prompt tuning for vision-language models via text feature dispersion. In ICLR. OpenReview.net, 2024. 10
Robustifying Vision-Language Models via Test-Time Prompt Adaptation
Zanella, M. and Ayed, I. B. On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning? In CVPR, pp. 23783–23793. IEEE, 2024.
Zhu, X., Zhu, B., Wang, S., Fang, J., Zhao, K., Zhang, H., and He, X. Principled steering via null-space projection for jailbreak defense in vision-language models. CoRR, abs/2603.22094, 2026d.
Zhang, J., Yi, Q., and Sang, J. Towards adversarial attack on vision-language pre-training models. In ACM Multimedia, pp. 5005–5013. ACM, 2022a.
Zhu, Y., Ji, Y., Zhao, Z., Wu, G., and Wang, L. AWT: transferring vision-language models via augmentation, weighting, and transportation. In NeurIPS, 2024c.
Zhang, J., Ma, X., Wang, X., Qiu, L., Wang, J., Jiang, Y., and Sang, J. Adversarial prompt tuning for vision-language models. In ECCV (45), volume 15103 of Lecture Notes in Computer Science, pp. 56–72. Springer, 2024. Zhang, M., Levine, S., and Finn, C. MEMO: test time robustness via adaptation and augmentation. In NeurIPS, 2022b. Zhao, S., Zhu, Q., Xiong, S., Ruan, S., Fan, Y., Duan, R., Guo, Q., and Wei, X. Enhancing adversarial robustness of vision language models via adversarial mixture prompt tuning. CoRR, abs/2505.17509, 2025. Zhao, Y., Pang, T., Du, C., Yang, X., Li, C., Cheung, N., and Lin, M. On evaluating adversarial robustness of large vision-language models. In NeurIPS, 2023. Zhu, X., Zhu, B., Tan, Y., Wang, S., Hao, Y., and Zhang, H. Enhancing zero-shot vision models by label-free prompt distribution learning and bias correcting. In NeurIPS, 2024a. Zhu, X., Zhu, B., Tan, Y., Wang, S., Hao, Y., and Zhang, H. Selective vision-language subspace projection for fewshot CLIP. In ACM Multimedia, pp. 3848–3857. ACM, 2024b. Zhu, X., Wang, S., Zhu, B., Li, M., Li, Y., Fang, J., Wang, Z., Wang, D., and Zhang, H. Dynamic multimodal prototype learning in vision-language models. In ICCV, pp. 2501– 2511. IEEE, 2025a. Zhu, X., Zhu, B., Wang, S., Zhao, K., and Zhang, H. Enhancing CLIP robustness via cross-modality alignment. CoRR, abs/2510.24038, 2025b. Zhu, X., Zhao, K., Yi, L., Wang, S., Wang, Z., Zhu, B., and Zhang, H. Look carefully: Adaptive visual reinforcements in multimodal large language models for hallucination mitigation. CoRR, abs/2602.24041, 2026a. Zhu, X., Zhu, B., Fang, J., Wang, S., Zhang, Y., Wang, X., and He, X. Guardalign: Test-time safety alignment in multimodal large language models. CoRR, abs/2602.24027, 2026b. Zhu, X., Zhu, B., Li, Y., Fang, J., Wang, S., Zhao, K., and Zhang, H. Hierarchical semantic alignment for image clustering. In AAAI, pp. 29177–29185. AAAI Press, 2026c. 11
Robustifying Vision-Language Models via Test-Time Prompt Adaptation
A. Proof of Theorem 1 Theorem 3.1 (Decomposition of Alignment Objective). The Optimal Transport objective (LOT ) imposes a stricter bound by decomposing into a centroid alignment term and a structural variance penalty: W22 (Pt , Qk ) ≈ | {z } LOT (RITA)
∥µ − µ ∥2 | x {z z }
+
Lmean (TPT-equivalent)
B2 (Σx , Σz ) | {z }
,
(13)
Rvar (Structural Penalty)
where B2 (A, B) = Tr(A + B − 2(A1/2 BA1/2 )1/2 ) is the Bures metric, quantifying the geometric mismatch between covariances. Proof. Let X ∼ µ and Z ∼ ν be random vectors in Rd with mean vectors µx , µz and covariance matrices Σx , Σz , respectively. The 2-Wasserstein distance is defined as the minimum expected transport cost over all valid joint couplings π ∈ Π(µ, ν): W22 (µ, ν) = inf E(X,Z)∼π ∥X − Z∥2 . (14) π∈Π(µ,ν)
We first expand the squared Euclidean cost by centering the variables around their respective means. Let X̃ = X − µx and Z̃ = Z − µz . The cost function can be rewritten as: ∥X − Z∥2 = ∥(X̃ − Z̃) + (µx − µz )∥2 = ∥µx − µz ∥2 + ∥X̃∥2 + ∥Z̃∥2 + 2(µx − µz )⊤ (X̃ − Z̃) − 2X̃ ⊤ Z̃.
(15)
Taking the expectation Eπ [·], the linear terms vanish because the variables are centered (i.e., E[X̃] = E[Z̃] = 0). Utilizing the identity E[∥X̃∥2 ] = Tr(Σx ), the expected cost simplifies to: Eπ ∥X − Z∥2 = ∥µx − µz ∥2 + Tr(Σx ) + Tr(Σz ) − 2Eπ X̃ ⊤ Z̃ . (16) To find W22 , we minimize Eq. (16) over the coupling π. Since the mean difference and trace terms are constants independent of π, the problem reduces to maximizing the correlation term Eπ [X̃ ⊤ Z̃]. For the family of elliptical distributions (e.g., Gaussians), it is a known result (Gelbrich, 1990) that the optimal coupling yields: 1/2 1/2 sup Eπ X̃ ⊤ Z̃ = Tr (Σ1/2 . (17) x Σz Σx ) π∈Π
Substituting this optimal correlation back into Eq. (16), we obtain: 1/2 1/2 W22 (µ, ν) = ∥µx − µz ∥2 + Tr(Σx ) + Tr(Σz ) − 2 Tr (Σ1/2 Σ Σ ) z x x 1/2 1/2 1/2 2 = ∥µx − µz ∥ + Tr Σx + Σz − 2(Σx Σz Σx ) . {z } | | {z } L mean
(18)
B2 (Σx ,Σz )
The first term corresponds to the centroid distance (Lmean ), and the second term is the squared Bures metric (B2 ), representing the structural variance cost. For general distributions, this expression serves as a tight lower bound, confirming that minimizing W22 inherently constrains both the first-order (mean) and second-order (variance) geometric moments. □
12
Robustifying Vision-Language Models via Test-Time Prompt Adaptation
Method
ImageNet Acc. Rob.
ImageNet-A Acc. Rob.
ImageNet-V2 Acc. Rob.
ImageNet-R Acc. Rob.
ImageNet-S Acc. Rob.
OOD Avg. Acc. Rob.
ViT-B/32
CLIP Ensemble TPT C-TPT MTA R-TPT RITA
62.0 64.4 63.6 63.5 64.9 64.4 64.8
0.7 52.3 36.6 33.4 40.1 52.1 52.7
29.5 34.1 34.5 30.5 37.7 36.9 35.4
0.1 21.4 9.3 7.6 11.1 21.9 22.5
54.7 58.1 56.9 55.9 58.3 58.0 58.5
1.5 45.6 30.4 27.4 33.2 45.5 45.9
66.2 64.5 69.1 67.0 70.3 67.5 65.7
6.9 55.2 49.4 45.0 52.3 55.8 56.3
40.8 39.2 41.6 41.8 43.4 41.7 40.8
4.5 31.2 30.4 30.1 31.5 31.2 32.3
47.8 49.0 50.5 48.8 52.4 51.0 50.1
3.2 38.3 31.2 28.7 32.0 38.6 39.3
ViT-B/16
Table 5. Results (%) of various adaptation methods on ImageNet and ImageNet-OOD datasets with ϵ = 1.0. OOD Avg. refers to the average results among four ImageNet-OOD datasets.
CLIP Ensemble TPT C-TPT MTA R-TPT RITA
66.7 68.8 68.9 68.1 69.0 69.1 69.1
0.6 54.4 42.4 38.0 44.4 54.4 55.1
47.7 55.8 54.7 49.7 57.3 57.2 55.8
0.1 33.6 14.9 11.3 17.5 34.7 35.0
60.8 62.8 63.6 61.9 63.4 63.5 63.2
0.2 47.4 35.7 31.4 37.2 48.0 48.4
73.9 72.9 77.1 74.8 76.9 75.5 74.0
3.5 62.7 57.3 51.9 58.9 63.7 64.1
46.1 46.5 47.9 47.2 48.4 47.7 47.1
2.2 35.1 35.6 34.4 35.8 36.5 37.2
57.1 59.5 60.8 58.4 61.5 60.9 60.0
1.5 44.7 37.1 3.4 38.7 45.7 46.2
B. Implementation Details For fair comparison, all approaches use the same pre-trained CLIP backbone and identical AugMix-based test-time augmentations, without external data, foundation models, or additional knowledge. We report average accuracy on clean samples and adversarial accuracy under PGD attacks with different perturbation budgets for default. Adversarial examples are generated on the original CLIP model, reflecting a realistic threat model. For most experiments, we adopt a default setting that utilizes class descriptors and sets the subset size |S| = 64. However, to accommodate the distinct characteristics of specific benchmarks, we adjust these parameters for several datasets as follows: 1) For EuroSAT, we disable descriptors for the ViT-B/16 backbone and set |S| = 32 for both ViT-B/16 and ViT-B/32; 2) For ImageNet-A and ImageNet-R, the size of subset |S| is 32; 3) For ImageNet-Sketch, descriptors are disabled, and the subset size |S| is 32. For all other datasets not explicitly mentioned, the default configuration is maintained. These refinements are designed to better align domain-specific semantic features with the test-time adaptation process.
C. Additional Results C.1. Results on ImageNet and ImageNet-OOD datasets. Table 5 presents the performance comparison across ImageNet and its four Out-of-Distribution (OOD) variants. The results demonstrate that RITA maintains a significant robustness advantage even when addressing large-scale general visual tasks. On the standard ImageNet, RITA achieves state-of-the-art or competitive accuracies across both architectures under both settings; for instance, it reaches 55.1% robustness on ViT-B/16, a substantial leap from the vanilla CLIP’s 0.6%. RITA’s performance is equally compelling on the more challenging OOD variants, where its average OOD robust accuracy reaches 39.3% and 46.2% in two backbones, outperforming other methods. These findings validate that RITA not only excels in fine-grained tasks but also provides robust protection against diverse adversarial threats in large-scale general scenarios and under various distribution shifts. C.2. Evaluation under enhanced adversarial attacks. Table 6 presents the performance of various test-time adaptation methods across eight fine-grained datasets under a more stringent adversarial constraint, where ϵ is set to 4.0. The experimental results indicate that as the attack intensity increases, the accuracy of the vanilla CLIP drops nearly to zero, whereas RITA demonstrates exceptional interference resistance. Specifically, our method achieves state-of-the-art results across both backbones, reaching 32.9% and 35.0% in average robust accuracy, respectively. These findings further confirm that RITA maintains high classification precision while exhibiting remarkable predictive stability, thereby validating its substantial practical value in mitigating complex and intense adversarial risks in real-world scenarios.
13
Robustifying Vision-Language Models via Test-Time Prompt Adaptation Table 6. Results (%) of adaptation methods on fine-grained classification datasets with ϵ = 4.0.
ViT-B/32
Caltech101 Pets Cars Flower102 Aircraft DTD EuroSAT UCF101 Avg. Acc. Rob. Acc. Rob. Acc. Rob. Acc. Rob. Acc. Rob. Acc. Rob. Acc. Rob. Acc. Rob. Acc. Rob.
CLIP 90.9 Ensemble 91.6 TPT 91.4 C-TPT 91.8 MTA 91.8 R-TPT 90.6 RITA 92.3
2.5 73.7 60.0 53.7 73.9 74.6 74.8
83.0 0.0 85.0 47.0 84.1 30.3 84.9 21.5 85.8 45.8 84.5 44.5 85.9 47.4
49.7 0.0 57.8 18.5 62.9 16.4 60.8 9.2 64.1 19.2 63.1 20.5 59.6 20.1
65.8 0.0 67.4 34.8 63.8 28.5 65.9 22.1 64.8 34.2 62.6 34.1 68.7 35.6
18.3 20.1 19.0 17.7 20.4 19.1 20.2
0.0 7.2 3.9 2.4 6.1 6.6 7.7
40.8 0.2 46.1 29.2 42.2 19.9 44.3 16.4 44.0 22.0 42.1 27.3 46.2 30.2
18.6 32.5 35.1 34.7 34.5 32.0 33.4
0.0 7.4 6.5 5.7 5.3 7.1 8.5
62.1 0.0 61.6 37.8 62.3 21.8 62.6 17.8 63.6 33.1 62.8 37.1 62.8 39.1
53.6 0.3 57.8 32.0 57.6 23.4 57.8 18.6 58.6 29.9 57.1 31.4 58.6 32.9
ViT-B/16
Method
CLIP 85.9 Ensemble 92.1 TPT 94.1 C-TPT 93.9 MTA 94.3 R-TPT 93.7 RITA 93.8
0.7 76.8 60.6 49.6 73.6 78.3 78.5
83.5 0.0 88.7 47.9 87.4 31.0 88.2 21.1 88.0 51.2 87.2 45.6 89.8 48.1
55.7 0.0 63.2 22.4 66.5 13.8 65.8 9.2 67.7 25.7 67.0 23.9 64.2 24.2
61.7 0.0 70.8 37.6 66.1 23.7 69.6 17.2 65.0 31.7 68.7 34.8 71.6 38.4
15.7 0.0 25.9 10.2 23.4 4.4 23.9 2.0 24.0 7.4 23.9 10.5 26.2 11.3
40.4 0.0 50.9 33.2 45.9 17.4 45.9 12.7 46.5 21.5 46.4 30.4 51.5 34.6
23.7 32.9 42.6 42.3 42.5 34.7 33.4
0.0 6.6 4.6 5.2 6.5 6.3 7.9
58.9 0.0 64.6 35.6 67.9 20.3 65.6 14.2 67.5 30.9 67.2 35.2 65.5 37.2
53.2 0.1 61.1 33.8 61.7 21.9 61.9 16.4 61.9 31.0 61.1 33.1 62.0 35.0
Table 7. Results (%) of adaptation methods on fine-grained classification datasets using RN50 with ϵ set to 1.0. Method
Caltech101 Pets Cars Flower102 Aircraft DTD EuroSAT UCF101 Avg. Acc. Rob. Acc. Rob. Acc. Rob. Acc. Rob. Acc. Rob. Acc. Rob. Acc. Rob. Acc. Rob. Acc. Rob.
CLIP 84.9 Ensemble 84.6 TPT 85.1 C-TPT 84.8 85.3 MTA R-TPT 86.7 RITA 85.7
2.6 78.2 7.0 3.7 65.9 78.2 79.3
83.5 0.0 85.1 75.4 84.7 0.1 83.6 0.0 84.8 59.8 84.6 74.2 86.3 77.4
53.7 0.0 52.6 38.4 54.4 0.0 55.6 0.0 55.7 17.8 56.1 38.6 54.2 39.9
61.7 0.0 65.4 56.2 62.1 0.0 64.8 0.0 61.0 31.5 60.6 51.9 66.2 56.9
14.7 0.0 15.7 11.8 15.3 5.2 16.7 7.2 15.9 10.3 16.4 11.8 16.9 12.4
40.4 0.8 43.1 38.2 42.4 4.3 41.5 1.3 40.3 18.8 41.3 33.5 43.8 39.5
18.7 0.0 22.0 14.6 22.4 0.0 22.2 0.0 22.5 1.6 21.2 15.1 20.2 15.9
57.5 0.0 56.3 49.5 60.2 0.3 60.1 0.1 60.6 31.3 59.5 49.2 58.8 50.0
51.8 0.4 53.1 45.2 53.3 2.1 53.6 1.5 53.2 29.6 53.3 43.0 54.0 46.4
Method Caltech101
Pets
Cars
adaptive
CLIP R-TPT RITA
30.4 87.3 88.2
4.9 76.3 79.5
7.2 55.3 57.8
3.2 60.8 66.2
0.2 17.4 19.1
2.4 35.3 37.6
0.0 20.5 23.8
0.8 52.2 54.8
6.1 50.6 53.4
AA
CLIP R-TPT RITA
13.2 87.9 89.8
4.9 78.0 82.0
0.3 51.4 53.7
2.6 59.4 61.8
0.0 20.3 22.6
0.0 41.5 47.7
0.0 24.2 26.0
4.6 58.2 59.9
3.2 52.6 55.4
FGSM
Table 8. Robust accuracy (%) on fine-grained classification datasets under different attacks using ViT-B/16 with ϵ = 1.0. Flower102 Aircraft
DTD
EuroSAT UCF101
Avg.
CLIP R-TPT RITA
6.2 84.8 85.9
2.4 73.6 74.5
0.5 43.6 44.2
0.0 54.3 59.4
0.0 19.9 21.7
0.4 36.2 42.1
0.0 23.1 24.4
1.8 50.3 51.8
1.4 48.2 50.5
C.3. Analysis on an alternative CLIP backbone. In Table 7, we further evaluate the performance of RITA using RN50 as the vision backbone to verify its generalizability across different architectures. The experimental results demonstrate that RITA exhibits superior robustness across all eight fine-grained classification datasets. Notably, despite RN50 having a relatively weaker baseline representation capability compared to the ViT series, RITA consistently outperforms other adaptation methods, while maintaining high clean accuracy. These findings provide strong evidence that our method delivers cross-architecture robustness gains and effectively mitigates the vulnerability of diverse vision encoders to adversarial attacks. C.4. Robustness evaluation under other attacks. We conduct experiments under various adversarial attack protocols, including adaptive attack that is aware of augmentation strategies, AutoAttack (AA), and FGSM. As reported in Table 8, RITA consistently achieves the highest robust accuracy 14
Robustifying Vision-Language Models via Test-Time Prompt Adaptation Table 9. Classification accuracy (%) on 10 datasets using EVA-CLIP and OpenCLIP backbones. Caltech101 Pets Cars Flower102 Aircraft DTD EuroSAT UCF101 SUN397 Food101 Avg. Acc. Rob. Acc. Rob. Acc. Rob. Acc. Rob. Acc. Rob. Acc. Rob. Acc. Rob. Acc. Rob. Acc. Rob. Acc. Rob. Acc. Rob.
Method
OpenCLIP 91.3 12.3 89.2 1.2 75.7 2.9 66.9 0.2 17.7 0.0 51.3 3.1 50.1 0.4 67.3 0.1 69.6 1.9 85.9 1.4 66.5 2.4 + RITA 92.4 89.9 91.7 78.2 78.4 50.2 73.2 62.4 25.1 19.3 55.9 46.3 51.5 29.9 69.6 53.5 75.2 54.0 88.3 60.7 70.1 54.4
original attacked views cache augmented views
3.9
KL Divergence
KL Divergence
EVA-CLIP 86.3 5.2 92.2 0.3 78.6 4.5 75.9 1.2 24.8 0.0 53.1 1.7 67.0 0.5 63.2 0.0 79.7 4.2 89.4 0.9 71.0 1.6 + RITA 87.1 84.6 93.1 80.4 79.5 48.2 78.6 64.8 27.5 17.5 56.2 41.4 69.8 38.2 64.9 50.8 82.3 59.2 91.2 62.4 73.0 54.8
3.8 3.7 3.6 0
10
20 30 Class Index
40
(a) DTD
4.7 4.6 4.5 4.4 4.3 4.2
original attacked views cache augmented views
0
20
40 60 80 Class Index
100
(b) Caltech101
Figure 8. KL divergence per class on (a) DTD and (b) Caltech101. Lower KL values signify superior vision-text alignment.
across all eight datasets and all attack types compared to the CLIP and R-TPT baselines. Specifically, under the more rigorous AutoAttack, RITA maintains an average robust accuracy of 55.4%, outperforming R-TPT by 2.8%. Notably, RITA shows significant gains on challenging datasets like DTD and EuroSAT across all attack settings. These results demonstrate that the defensive capability of RITA is not tailored to a specific attack but generalizes well to diverse adversarial threats, confirming its potential for securing VLMs in various hostile environments. C.5. Generalization to Other VLM Backbones. To further validate the architectural agnosticity of RITA, we extend our evaluation to other VLMs, including OpenCLIP and EVA-CLIP. As shown in Table 9, we expanded RITA to 10 datasets by incorporating SUN397 and Food101. The results consistently demonstrate that RITA significantly boosts adversarial robustness across all architectures. For instance, when applied to OpenCLIP, RITA improves the average robust accuracy from 2.4% to 54.4%. On the newly added SUN397 and Food101 datasets, RITA achieves substantial gains, reaching robust accuracies of 54.0% and 60.7% for OpenCLIP, and 59.2% and 62.4% for EVA-CLIP, respectively. These findings underscore that the robustness gains of RITA are consistent across diverse VLM backbones and broader dataset distributions, reinforcing its effectiveness as a general test-time adaptation framework.
D. Verification of the importance of the cache mechanism D.1. Ablation study of the dynamic cache. In Table 10, we conduct an ablation study to specifically validate the significance of the cache mechanism dcOT within the RITA framework. The results indicate that while utilizing the cache mechanism in isolation yields lower standard accuracy (39.7% on fine-grained and 20.2% on ImageNet) due to the absence of real-time alignment, it consistently outperforms the vanilla CLIP baseline in robustness metrics, achieving 27.1% and 29.6% respectively, compared to CLIP’s 4.7% and 3.2%. This evidence underscores that the historical priors preserved in the cache serve as an essential reference for stabilizing predictions under adversarial perturbations. Most importantly, the synergy between the cache mechanism and the optimal transport module dOT leads to peak performance across all metrics, notably boosting fine-grained robustness from 49.4% to 55.0%. This further demonstrates that the cache provides critical semantic supplementation to the multimodal alignment, establishing a more robust and historically-aware inference framework.
15
Robustifying Vision-Language Models via Test-Time Prompt Adaptation Table 10. Main component analysis (%) on fine-grained datasets and ImageNet dataset using ViT-B/32 with ϵ = 1.0.
dOT
dcOT
Fine-grained Acc. Rob.
ImageNet Acc. Rob.
% " % "
% % " "
53.6 57.0 39.7 57.9
47.8 63.9 20.2 64.8
4.7 49.4 27.1 55.0
3.2 52.3 29.6 52.7
Algorithm 1 RITA: Robust test-tIme prompT Adaptation (m) 1: Input: Test stream {x̂t }, encoders Φimg , Φtext , text prompts {zk }M m=1 , entropy threshold γ, max cache size per class Nk , cache weight α. (m) m 2: Initialize: Empty cache {X̂k }K k=1 ← ∅; extract text features zk = Φtext (zk ) and construct text distributions P M 1 m Qk = M m=1 δzk . 3: for each test image x̂t do n n 4: Generate N augmented views {x̂nt }N n=1 , extract visual features xt = Φimg (x̂t ), and construct visual distribution PN 1 Pt = N n=1 δxnt . Cache Update: For each view x̂nt with entropy H(pnt ) < γ, get pseudo-label k̂ = argmax pnt (k). Add xnt to X̂k̂ (if 5: k
6: 7: 8: 9: 10: 11:
|X̂k̂ | ≥ Nk and H(pnt ) is lower, replace the max-entropy sample). for class k = 1 to K do Compute global OT distance dOT (Pt , Qk ; Ct,k ) where Ct,k (n, m) = 1 − cos(xnt , zm k ). ∗ If X̂k ̸= ∅, align features X̃k = X̂k Wk to build cache dist Q̃k , and calculate cache OT distance dcOT (Pt , Q̃k ; C̃t,k ) Else dcOT = 0. end for Output: Predicted label ŷ = argmin dOT (Pt , Qk ; Ct,k ) + αdcOT (Pt , Q̃k ; C̃t,k ) . k∈[K]
12: end for
D.2. Semantic alignment analysis of the cache mechanism. To demonstrate the effectiveness of the cache mechanism, we extract the augmented view features from the DTD and Caltech101 datasets and compare them with the original unaugmented visual features from a ”vision-text” alignment perspective. Specifically, for each class, we first compute the class-conditional assignment distribution over all text prototypes, which reflects how the visual features of each class are semantically aligned with the textual features. Subsequently, we measure the KL divergence between this empirical distribution and an ideal one-hot target distribution that assigns all probability mass to the ground-truth class. A lower KL value indicates stronger vision-text alignment. As illustrated in Figure 8, we present the scatter plots of KL divergence for all classes in DTD and Caltech101. Each point represents the vision-text alignment quality of a specific category, where red dots denote original adversarial views and green dots represent selected augmented views utilized by our cache mechanism. The KL divergence of original views exhibits high variance and remains at an elevated level, reflecting severe adversarial bias. Upon introducing the cache mechanism, the green dots show both a downward shift and reduced dispersion. This indicates that the augmented views effectively calibrate the corrupted feature representations by aggregating historical priors, pulling the model closer to the ideal one-hot distribution. Even for categories where the original KL divergence is particularly high, the augmented features still achieve significant alignment gains. This cross-category consistency provides a granular foundation for the superior robustness of the RITA framework when encountering diverse adversarial attacks.
E. Algorithm for RITA Algorithm 1 summarizes the RITA framework. RITA extracts features from augmented views, aligns visual-textual distributions via optimal transport, and maintains a dynamic cache for progressive refinement.
16