arXiv:2604.09532v1 [cs.CV] 10 Apr 2026
Seeing is Believing: Robust Vision-Guided Cross-Modal Prompt Learning under Label Noise Zibin Geng∗
Xuefeng Jiang∗
Jia Li
Institute of Computing Technology, Chinese Academy of Sciences University of Chinese Academy of Sciences Beijing, China [email protected]
Institute of Computing Technology, Chinese Academy of Sciences University of Chinese Academy of Sciences Beijing, China [email protected]
Institute of Information Engineering, Chinese Academy of Sciences University of Chinese Academy of Sciences Beijing, China [email protected]
Zheng Li
Tian Wen
Lvhua Wu
PCALab, VCIP, College of Computer Science, Nankai University Tianjin, China [email protected]
Institute of Computing Technology, Chinese Academy of Sciences University of Chinese Academy of Sciences Beijing, China [email protected]
Institute of Computing Technology, Chinese Academy of Sciences University of Chinese Academy of Sciences Beijing, China [email protected]
Sheng Sun
Yuwei Wang
Min Liu†
Institute of Computing Technology, Chinese Academy of Sciences Beijing, China [email protected]
Institute of Computing Technology, Chinese Academy of Sciences Beijing, China [email protected]
Institute of Computing Technology, Chinese Academy of Sciences Beijing, China [email protected]
Abstract Prompt learning is a parameter-efficient approach for vision-language models, yet its robustness under label noise is less investigated. Visual content contains richer and more reliable semantic information, which remains more robust under label noise. However, the prompt itself is highly susceptible to label noise. Motivated by this intuition, we propose VisPrompt, a lightweight and robust vision-guided prompt learning framework for noisy-label settings. Specifically, we exploit a cross-modal attention mechanism to reversely inject visual semantics into prompt representations. This enables the prompt tokens to selectively aggregate visual information relevant to the current sample, thereby improving robustness by anchoring prompt learning to stable instance-level visual evidence and reducing the influence of noisy supervision. To address the instability caused by using the same way of injecting visual information for all samples, despite differences in the quality of their visual cues, we further introduce a lightweight conditional modulation mechanism to adaptively control the strength of visual information injection, which strikes a more robust balance between text-side semantic priors and image-side instance evidence. The proposed framework effectively suppresses the noise-induced disturbances, reduce instability in prompt updates, and alleviate memorization of mislabeled samples. VisPrompt significantly improves robustness while keeping the pretrained VLM backbone frozen and introducing only a small amount of additional trainable parameters. Extensive experiments under synthetic and real-world label noise demonstrate ∗ Both authors contributed equally to this research. † Corresponding author.
that VisPrompt generally outperforms existing baselines on seven benchmark datasets and achieves stronger robustness. Our code is publicly available at https://github.com/gezbww/Vis_Prompt.
CCS Concepts • Computing methodologies → Computer vision tasks; Object recognition.
Keywords Prompt Learning, Vision-Language Models, Label Noise
1
Introduction
Vision-language models (VLMs) [14, 29, 41] pretrained on largescale image-text corpora have exhibited remarkable transferability, enabling strong zero-shot performance across a wide range of downstream recognition and retrieval tasks. Adapting such pretrained VLMs to a specific task or domain still requires a lightweight interface that is both data-efficient and parameter-efficient. Prompt learning has therefore become a widely adopted paradigm: It keeps the pretrained backbone frozen and optimizes only a small set of learnable context tokens, often achieving competitive performance with minimal trainable overhead [15, 25, 27, 30, 44, 55]. Despite this efficiency, prompt learning remains highly sensitive to annotation quality in practice [55]. Real-world datasets often contain mislabeled samples, and such corrupted supervision can directly mislead the optimization of soft prompts.
Conference’17, July 2017, Washington, DC, USA
Text Encoder
Output Layer
This is a cat
Hidden Layers
Image Encoder
Trovato et al.
Noisy Label Wrong Feature
Cat
Right Feature
Cat Dog
Prior methods
Output Layer
FiLM
Ours
Text Encoder
Hidden Layers
This is a cat
Noisy Label
Muti-head Attention
Image Encoder
Wrong Feature Right Feature
Dog Cat Dog
Figure 1: Prior label-driven prompt methods can be misled by noisy supervision and produce incorrect predictions, while our method injects reliable image-grounded information to guide prompt learning toward the correct class.
This issue is especially pronounced in prompt learning: Since the image and language encoders are frozen, the burden of task adaptation is concentrated on a small number of prompt embeddings, making them highly exposed to noisy gradients induced by incorrect labels. As the noise rate increases, the learned prompts can gradually drift toward spurious class semantics, leading to unstable optimization and degraded generalization. Recent studies have shown that prompt learning with frozen VLM backbones exhibits a certain degree of tolerance to label noise, which has motivated a growing body of research on noise-robust prompt adaptation [9, 47]. Existing methods [9] mainly improve robustness from three directions, including confidence-based noise detection and trust estimation [46, 52], robust objective design [13, 35], and sample refinement strategies [35]. These approaches have achieved encouraging results, but most of them still tackle noisy supervision mainly through label-centric mechanisms, rather than rethinking prompt optimization from the perspective of visual guidance. This design pattern leaves a more reliable source of information underexplored. In contrast, even under corrupted annotations, large-scale pretrained VLMs such as CLIP [41] can still provide rich instance-level semantic cues from images, offering a more stable source of guidance under label noise. However, current methods do not fully leverage such visual cues as a primary anchor for prompt optimization. Instead, they often rely on early-stage confidence estimation or sample discrimination to identify clean data under noisy supervision. Once these early decisions are inaccurate, the resulting confirmation bias can be inherited by subsequent optimization, causing prompt updates to be guided by increasingly unreliable supervision. Motivated by this intuition, we propose an instance-grounded semantic anchoring principle for prompt learning in the presence of label noise. Prompt adaptation should be anchored to the semantic content of the input image, rather than relying solely on the observed noisy label. This principle is based on a simple intuition that, although annotations may be corrupted, the image itself still preserves instance-specific semantic evidence that can provide a more stable guidance signal for prompt optimization.
To instantiate this principle, we propose VisPrompt (VisionGuided Cross-Modal Prompt Learning under Label Noise), a robust prompt learning framework that explicitly injects image-grounded semantics into prompt adaptation in noisy-label settings. VisPrompt first introduces cross-modal attention to inject visual evidence into the learnable prompts. Instead of updating prompts solely according to label supervision, our method allows prompt tokens to selectively aggregate semantically relevant information from the images. In this way, the prompt representation is no longer optimized only toward the observed noisy objective, but is also constrained by the semantics of the current instance. As illustrated in Fig. 1, such image-grounded conditioning helps steer the prediction toward the correct class even when the assigned label is corrupted. However, the usefulness of visual evidence is not uniform across samples. Applying the same fusion strength to all instances may underuse highly informative visual cues, while also introducing unstable or irrelevant perturbations from low-quality ones. To address this issue, we further introduces FiLM (Feature-wise Linear Modulation) as an instance-adaptive control mechanism. Conditioned on the current visual representation, FiLM adaptively modulates how much visual evidence should be injected into the prompt and which feature dimensions should be emphasized or suppressed. This selective modulation enables a more controlled integration of image-side evidence, leading to more stable prompt updates under noisy supervision. From a robustness perspective, cross-modal attention and FiLM modulation selectively introduce and regulate visual evidence, thereby improving the stability of prompt optimization under noisy supervision. Importantly, VisPrompt improves robustness with minimal trainable overhead, introducing less than 1% additional parameters while keeping the pretrained VLM backbone frozen. Our main contributions are summarized as follows: • We propose VisPrompt, a lightweight and robust prompt learning framework for noisy-label settings. It reformulates prompt adaptation from a cross-modal perspective and exploits imagegrounded semantics to guide prompt learning. This reduces the interference of corrupted labels during optimization. • We introduce a vision-guided prompt modulation mechanism that combines cross-modal attention with FiLM gating. Crossmodal attention extracts visual evidence that is more informative and reliable. FiLM then adaptively controls how this evidence is injected into the prompt representation. This enables selective enhancement of reliable cues and suppression of unstable perturbations. • We validate the robustness of VisPrompt through theoretical analysis. The analysis shows that cross-modal attention yields a denoised approximation of clean semantics. It also shows that FiLM injects such evidence into prompts in a bounded and stable manner. Extensive experiments further demonstrate that VisPrompt achieves competitive or superior performance compared with representative robust prompt learning based methods on seven datasets with synthetic and real-world label noise.
2 Related Work 2.1 Prompt Learning With the rapid advance of vision-language models, prompt learning emerges as a pivotal research direction as a parameter-efficient
Seeing is Believing: Robust Vision-Guided Cross-Modal Prompt Learning under Label Noise
learning approach. Numerous studies [1, 2, 41] demonstrate its efficiency and scalability, with CLIP [41] being the most representative work. Early prompt learning relies on manually designed templates, such that minor changes could result in significant performance variation. CoOP [55] introduces learnable continuous context on the text side and optimizes end to end with a frozen backbone, significantly improving adaptation efficiency. CoCoOp [54] leverages image conditioned context to enhance generalization to unseen classes, and MaPLe [20] extends prompt integration to deeper layers, coupling it along the entire vision-language pathway. KAPT [19] and ATPrompt [28] introduce external attribute knowledge and generic attribute tokens, respectively, to strengthen cross-domain generalization and class alignment. While these methods improve accuracy and transferability given clean supervision, their capability to handle noisy labels remains rarely explored.
2.2
Noisy Label Learning
In real-world datasets, label noise is inevitable, and corrupted annotations can cause severe performance degradation. To address this issue, numerous previous works have been proposed from diverse aspects, including robust loss functions [7, 8, 18, 31], robust regularization [12, 17, 32, 48], sample selection [18, 46] and meta-learning [26, 38, 43], loss correction [3, 49, 51], and robust training framework design [16, 23, 50]. However, these works mainly focus on the unimodal tasks like image or text classification. In prompt learning, however, research on handling noisy labels remains limited. PTNL [47] opens this direction by revealing the robustness potential of prompt learning. GCE [53] enhances the robustness of prompt learning from the perspective of loss function design. JoAPR [9] fits a two-component Gaussian mixture to the loss distribution and uses adaptive thresholds for clean-sample selection and relabeling. NLPrompt [35] employs optimal transport to partition clean and noisy subsets, while TrustCLIP [52] estimates trustworthiness from the structural semantics learned during training. Although these methods achieve meaningful gains, they mostly rely on auxiliary confidence modeling or sample partition strategies, and largely optimize robustness from the text or label side. As a result, they do not fully exploit image content itself as a relatively stable and instance-specific source of supervision under label corruption.
2.3
Vision-guided prompt generation
Beyond text-only prompt learning methods such as CoOp [55], recent studies have explored incorporating visual cues into prompt learning for adapting vision-language models. CoCoOp [54] conditions prompts on individual input images, while subsequent methods such as DPT [42], further strengthen cross-modal interaction by jointly leveraging visual and textual information. Other works, such as GalLoP [22] , exploit local visual evidence or attribute-level semantics to improve prompt quality and generalization. These studies collectively suggest that visual guidance can enrich prompt learning with instance-specific semantics and improve adaptation performance. While existing approaches have mitigated label noise to some extent, they predominantly focus on loss function design, noisy-sample detection, and data purification, often overlooking the potential of visually guided prompt generation. To bridge this gap,
Conference’17, July 2017, Washington, DC, USA
we propose VisPrompt, a vision-guided framework for few-shot learning under noisy label conditions.
3
Preliminary
We first introduce the standard prompt learning paradigm for visionlanguage pre-trained models. Given an input image, the image encoder and text encoder project visual and textual inputs into a shared embedding space, where classification is performed according to their similarity. For Vision Encoder, given an input image 𝐼 , the image encoder first divides it into 𝑀 patches and maps them into patch embeddings, 0 }. A learnable class token 𝑣 0 is then denoted by 𝑉 0 = {𝑣 10, . . . , 𝑣 𝑀 cls appended to the patch sequence. The resulting tokens are fed into a stack of vision transformer layers: [𝑉 𝑙 , 𝑣 𝑙cls ] = Image𝑙 [𝑉 𝑙 −1, 𝑣 𝑙cls−1 ] , 𝑙 = 1, . . . , 𝐿𝐼 . where 𝐿𝐼 denotes the number of layers in the visual encoder. The final image representation in the shared vision-language space is obtained by 𝐿𝐼 projecting the class token from the last layer: ℎ = 𝑓Proj 𝑣 cls . For Text Encoder, given a text prompt, the text encoder tok0 }. These token enizes it into 𝑁 tokens, denoted by 𝐶 0 = {𝑐 10, . . . , 𝑐 𝑁 embeddings are processed by a stack of text transformer layers: 𝑇 𝑙 = Text𝑙 (𝐶 𝑙 −1 ), 𝑙 = 1, . . . , 𝐿𝑇 , where 𝐿𝑇 is the number of layers in the text encoder. The final text feature is produced by projecting the last token at the top layer into the shared vision-language space: 𝐿𝑇 𝑔 = 𝑓Proj 𝑐 𝑁 . In standard prompt learning, the parameters of both the image encoder and the text encoder are kept frozen, and only a small set of learnable context tokens is optimized. For a class name represented by [CLASS], the prompt is constructed by combining it with 𝑛 learnable context tokens, denoted by 𝑃 = {𝑝 1, . . . , 𝑝𝑛 }. The resulting prompt can be formulated as C𝑘 = [𝑝 1, 𝑝 2, . . . , 𝑝𝑛 , class𝑘 ],
(1)
where class𝑘 denotes the tokenized name of the 𝑘-th class. The prompt is then fed into the text encoder to obtain the corresponding class-level text feature: 𝑔𝑘 = TextEncoder(C𝑘 ),
𝑘 = 1, . . . , 𝐾,
(2)
where 𝐾 is the number of classes. Given an input image, the image encoder produces image feature ℎ, while the text encoder generates a text prototype 𝑔𝑘 for each class. Result is performed in the shared embedding space according to the similarity between ℎ and 𝑔𝑘 : exp sim(ℎ, 𝑔𝑘 )/𝜏 𝑝 (𝑦 = 𝑘 | 𝐼 ) = Í𝐾 (3) , 𝑗=1 exp sim(ℎ, 𝑔 𝑗 )/𝜏 where sim(·, ·) denotes cosine similarity and 𝜏 is a temperature parameter. The model is trained by updating the learnable prompt tokens such that the text representation of the correct class becomes better aligned with the visual representation of the input image.
4
Methodology
To improve the robustness of prompt learning under noisy supervision, we propose VisPrompt, a visual-guided prompt learning framework for noise-robust adaptation. As illustrated in Fig. 2, the framework consists of two key components: (i) Cross-modal Visual Prompt Conditioning, which injects multiple local visual tokens
Conference’17, July 2017, Washington, DC, USA
Trovato et al.
step 1:Initialization
�2 �3
·
+
�n
Frozen
�1 �2 �3
�n
��
�� − �
� = � ∙ ��� + (� − �) ∙ ����
⊙ multiplication
OT Module
...
Trainable
��
�1
+ +1
step 4:Loss selection
·
FFN
Forward
�
��
Layer Norm
�n
�
��
step 3:FiLM modulation
...
...
�3
Sigmoid
�2
Linear
dog
�1
� Linear
A photo of
Text Encoder
Layer Norm
Text
�
Linear
�n
Image
��
Linear
...
�3
Cross-Modal Attention
�2
Layer Norm
Image Encoder
�1
step 2:Cross-modal attention
Certain
����
ℬ�
ℬ�
���
�
Uncertain
Figure 2: The overall architecture of VisPrompt which consists of four steps. Step 2 and Step 3 denote the robust update process. into the prompt context through cross-modal attention; (ii) FiLMbased Robust Modulation, which adaptively controls the injected visual guidance through conditional modulation and residual gating. By combining reliable image-grounded semantics with the generic prior of the text branch, VisPrompt mitigates noise-induced prompt drift and yields more stable and robust prompt optimization.
4.1
VisPrompt Framework
VisPrompt is instantiated through four sequential stages: Initialization, cross-modal visual prompt conditioning, FiLM-based robust modulation, and robust loss selection. 4.1.1 Learnable Context Initialization. Within the prompt learning framework, we introduce 𝑛 ctx learnable context tokens as the optimizable part of the prompt. Depending on the parameterization strategy, the context can be either class-shared or class-specific. For the class-shared setting, the learnable context is denoted as C ∈ R𝑛ctx ×𝑑 ,
(4)
where 𝑑 is the dimensionality of the text embedding space. For the class-specific setting, the context can be extended as C ∈ R𝑁 ×𝑛ctx ×𝑑 ,
(5)
where 𝑁 denotes the number of classes. 4.1.2 Visual Feature Projection. Unlike class labels, which may be corrupted at the sample level, local image content still preserves fine-grained semantic evidence from the underlying visual instance. To explicitly exploit such relatively reliable instance-level cues, we extract a set of local visual representations from the input image. Given an input image 𝑥𝑖 , the image encoder produces V𝑖 ∈ R𝑀 ×𝑑 𝑣 ,
Z𝑖 = V𝑖 W𝑝 ∈ R𝑀 ×𝑑 .
(8)
Projecting these tokens into the text-aligned space allows the model to access image-grounded semantics that are less affected by annotation errors. As a result, the prompt is not forced to rely solely on potentially misleading supervision, but can instead condition on more reliable instance-level cues. 4.1.3 Cross-modal Visual Prompt Conditioning. To reduce the direct influence of noisy labels on prompt updates, this module is designed to introduce an instance-specific image-conditioned signal into the learnable context. Instead of letting the prompt be updated solely by the supervisory signal of the assigned label, we explicitly establish cross-modal interactions between the context tokens and the projected visual tokens, so that the prompt can be conditioned on the semantic content of the current image. Let C denote the context tokens and Z𝑖 denote the projected visual token sequence associated with image 𝑥𝑖 . The cross-modal conditioning feature is defined as A𝑖 = MHA LN(C), LN(Z𝑖 ), LN(Z𝑖 ) , (9) where MHA(·) and LN(·) denote multi-head attention and layer normalization, respectively. Here, A𝑖 is constructed as the cross-modal conditioning representation for the current sample. Its role is to use the context tokens as queries to selectively retrieve semantically relevant local evidence from the visual tokens, and to organize such image-grounded information into a conditioning signal for subsequent prompt modulation.
(6)
where 𝑀 is the number of visual tokens and 𝑑 𝑣 is the dimensionality of the visual feature space. Since the visual and textual branches generally reside in different representation spaces, we employ a learnable linear projection matrix W𝑝 ∈ R𝑑 𝑣 ×𝑑
to align visual features with the prompt embedding space, yielding
(7)
4.1.4 FiLM-based Robust Modulation. Although visual evidence is generally more reliable than noisy labels, its quality may still vary across instances. To incorporate such instance-dependent visual cues in a controlled manner, we employ a FiLM-based modulation mechanism together with token-wise gating. Based on the crossmodal feature A𝑖 , we first generate FiLM modulation parameters
Seeing is Believing: Robust Vision-Guided Cross-Modal Prompt Learning under Label Noise
to perform feature-wise adjustment on the normalized context: C̃𝑖 = LN(C) ⊙ 1 + 𝜙𝛾 (A𝑖 ) + 𝜙 𝛽 (A𝑖 ), (10) and further compute a token-wise gate as G𝑖 = 𝜎 𝜙𝑔 (A𝑖 ) ,
(11)
where 𝜙𝛾 (·), 𝜙 𝛽 (·), and 𝜙𝑔 (·) are learnable mappings for the scaling term, bias term, and gate generation, respectively, 𝜎 (·) denotes the sigmoid function, and ⊙ denotes element-wise multiplication. Given the modulated context C̃𝑖 and the token-wise gate G𝑖 , we then perform a residual gated update: C𝑖′ = C + G𝑖 ⊙ C̃𝑖 − C , (12) followed by a feed-forward refinement: Ĉ𝑖 = C𝑖′ + FFN LN(C𝑖′ ) ,
Conference’17, July 2017, Washington, DC, USA
where U (𝑎, 𝑏) denotes the set of admissible transport plans with prescribed marginals 𝑎 and 𝑏, 𝜀 is the entropic regularization coefficient, and 𝐻 (Π) denotes the entropy term. The optimal transport plan Π ∗ is efficiently computed by the Sinkhorn algorithm. Based on Π ∗ , a reliability score is assigned to each sample. For sample 𝑖, the OT-induced pseudo-label is first defined as 𝑦ˆ𝑖 = arg max Π ∗𝑗𝑖 . The corresponding confidence score is then defined as 𝑟𝑖 = Π∗𝑦ˆ𝑖 𝑖 .
(13)
4.1.5 Robust Loss Design. The impact of label noise varies across samples. Clean samples usually provide relatively reliable supervision, whereas noisy samples are more likely to be dominated by corrupted annotations. When all samples are optimized with the same loss function, the model may gradually overfit noisy labels and thus lose robustness. To alleviate this issue, this work introduces an Optimal Transport (OT) [5] mechanism to estimate sample reliability under noisy supervision. According to the estimated reliability, the training set is partitioned into a reliable subset and an unreliable subset. Different loss functions are then assigned to these two subsets. This strategy allows the optimization process to better adapt to the heterogeneous quality of supervisory signals. The central function of OT is to determine a global transport plan between two predefined distributions under a given transport cost and marginal constraints. By jointly modeling all sample class matching relationships, OT can suppress locally abnormal assignments and produce a more globally consistent correspondence structure. Assume that the training set consists of 𝑁 samples and 𝐿 classes. The frozen image encoder extracts the image feature matrix 𝑉 ∈ R𝑁 ×𝑑 , where 𝑑 denotes the feature dimension, while the text encoder produces the class text feature matrix 𝑇 ∈ R𝐿×𝑑 . The similarity score between each sample and each class is then computed. The normalized similarity is further converted into the transport cost used in OT. Specifically, the normalized similarity between sample 𝑖 and class 𝑗 is defined as exp sim(𝑡 𝑗 , 𝑣𝑖 )/𝜏 𝑆 𝑗𝑖 = Í𝐿 , (14) 𝑘=1 exp(sim(𝑡𝑘 , 𝑣 𝑖 )/𝜏) where 𝜏 is the temperature coefficient and sim(·, ·) denotes the similarity function. The corresponding matching cost is defined by (15)
The transport plan is obtained by solving the entropic OT problem Π ∗ = arg
min ⟨Π, 𝐷⟩ − 𝜀𝐻 (Π),
Π∈ U (𝑎,𝑏 )
(18)
A sample is regarded as reliable if its OT assignment is consistent with the observed label and its confidence exceeds a predefined threshold 𝛿. Accordingly, the reliable subset and unreliable subset are defined as
The resulting Ĉ𝑖 is used as the image-conditioned prompt context for the current sample.
𝐷 𝑗𝑖 = − log 𝑆 𝑗𝑖 .
(17)
𝑗
(16)
B𝑥 = {(𝑥𝑖 , 𝑦𝑖 ) | 𝑦ˆ𝑖 = 𝑦𝑖 , 𝑟𝑖 ≥ 𝛿 } .
(19)
B𝑢 = {(𝑥𝑖 , 𝑦𝑖 ) | 𝑦ˆ𝑖 ≠ 𝑦𝑖 or 𝑟𝑖 < 𝛿 } .
(20)
The reliable subset is optimized with standard cross-entropy loss, ∑︁ 1 log 𝑝𝑖,𝑦𝑖 , (21) LCE = − |B𝑥 | (𝑥𝑖 ,𝑦𝑖 ) ∈ B𝑥
where 𝑝𝑖,𝑦𝑖 denotes the predicted probability of sample 𝑖 on its annotated class 𝑦𝑖 . For the unreliable subset, the generalized crossentropy loss is adopted to reduce the adverse effect of corrupted labels: 𝑞 ∑︁ 1 − 𝑝𝑖,𝑦𝑖 1 , 𝑞 ∈ (0, 1]. (22) LGCE = |B𝑢 | 𝑞 (𝑥𝑖 ,𝑦𝑖 ) ∈ B𝑢
The overall robust training objective is therefore formulated as Lrobust = 𝛼 LCE + (1 − 𝛼)LGCE,
(23)
where 𝛼 controls the extent of the unreliable subset for optimization. Remark. The above design preserves parameter-efficient for prompt learning: The pre-trained image encoder and text encoder remain frozen, while only the learnable prompt context and the lightweight visual-guided modulation modules are optimized.
4.2
Theoretical Analysis
Herein we explain why VisPrompt is robust under corrupted supervision from two aspects: Cross-modal attention can extract reliable semantic signals grounded in image content, while FiLM-based gated modulation injects this signal into the prompt in a controlled manner rather than uncontrolled prompt drift. 4.2.1 Basic assumptions. For each image 𝑥𝑖 , let 𝑠𝑖 denote the latent clean semantic representation of the instance in the shared visionlanguage space. Assume that: (i) among the 𝑀 visual tokens, there exists an informative subset whose elements are close to 𝑠𝑖 ; (ii) cross-modal attention assigns larger scores to informative tokens than to irrelevant ones, with a positive margin Δ𝑖 ; (iii) the FiLM and gating mappings are Lipschitz continuous [45], and the residual gate is bounded in [0, 1]. These assumptions are consistent with the roles of shared image-text representation, instance-conditioned prompts, and feature-wise modulation in prior studies [40, 54].
Conference’17, July 2017, Washington, DC, USA
Trovato et al.
4.2.2 Cross-modal semantic aggregation. The cross-modal attention output is formulated as 𝐴𝑖 =
𝑀 ∑︁ 𝑚=1
𝛼𝑖,𝑚 𝑧𝑖,𝑚 ,
exp(𝑎𝑖,𝑚 ) , 𝛼𝑖,𝑚 = Í𝑀 𝑟 =1 exp(𝑎𝑖,𝑟 )
(24)
where 𝑎𝑖,𝑚 is the attention score between the prompt query and the 𝑚-th visual token. Since informative tokens are assumed to have a score margin over irrelevant ones, the softmax weights concentrate on semantically relevant local evidence. As a result, the aggregated feature 𝐴𝑖 serves as a denoised approximation of the clean instance semantics 𝑠𝑖 , and the approximation error decreases as the attention margin increases. 4.2.3 Controlled prompt modulation. Let the FiLM-based robust modulation be denoted compactly by Ĉ𝑖 = F (C, 𝐴𝑖 ),
(25)
where F summarizes the FiLM transformation, token-wise gating, residual update, and FFN refinement defined in Eqs. (12)–(15). Denote by Ĉ𝑖★ = F (C, 𝑠𝑖 ) the ideal prompt that would be obtained if the clean semantic signal 𝑠𝑖 were directly available. Because FiLM is feature-wise and the residual gate is bounded, F is a stable Lipschitz mapping with respect to its conditioning signal, which means that perturbations in 𝐴𝑖 cannot be arbitrarily amplified during prompt modulation.
decision is preserved as long as the perturbation induced by noisy supervision stays below the clean margin.
5 Experiments 5.1 Datasets with Noisy Labels We evaluate our method on six benchmark datasets with synthetic symmetric label noise: EuroSAT [11], Flowers102 [34], OxfordPets [36], DTD [4], UCF101 [39], and Caltech101 [6]. These datasets cover diverse recognition objectives, and their statistics are summarized in Table 1. Owing to their reliable annotations, they provide a suitable testbed for controlled synthetic noise injection under the few-shot setting. Meanwhile, we further conduct evaluation on Food101N [24], a real-world noisy dataset collected from web sources. Unlike synthetic corruption, its label noise arises naturally from weak supervision and imperfect data collection. Regarding the synthetic label noise patterns, we introduce both symmetric and asymmetric label noise. For symmetric noise, each class label is flipped to any other class with equal probability. In contrast, asymmetric noise assigns different flip probabilities to each class, better mirroring real-world conditions. We consider six label noise rates at 12.5%, 25%, 37.5%, 50%, 62.5%, and 75% to provide broad coverage of plausible noise rates.
Table 1: Dataset Statistics.
Theorem 1. Under the above assumptions, there exists a constant 𝐿mod > 0 such that ∥ Ĉ𝑖 − Ĉ𝑖★ ∥ ≤ 𝐿mod 𝜀 𝑣 + 𝑐𝑖 𝑒 −Δ𝑖 , (26) where 𝜀 𝑣 measures the approximation error between informative visual tokens and the clean instance semantics, and 𝑐𝑖 𝑒 −Δ𝑖 is the residual distraction term induced by irrelevant tokens. Proof The proof follows two steps. First, by the positive attention margin, the softmax mass assigned to irrelevant visual tokens decays exponentially with Δ𝑖 , so the aggregated feature 𝐴𝑖 remains close to the clean semantic signal 𝑠𝑖 . Second, since FiLM and gate generation are Lipschitz [45] and the residual gate is bounded, the modulation map M transmits this perturbation in a controlled way rather than amplifying it, which directly yields Eq. (26). 4.2.4 Implication for robustness. Eq. (26) shows that the deviation of the image-conditioned prompt from its ideal clean-semantic counterpart is jointly controlled by two factors: the quality of informative local visual evidence and the attention margin over irrelevant tokens. Therefore, compared with purely label-driven prompt updates, the proposed design is less sensitive to corrupted supervision: Cross-modal attention suppresses noisy or irrelevant evidence at the aggregation stage, while FiLM-based gated modulation prevents such perturbations from causing excessive prompt drift. Furthermore, if the clean-prompt logit margin is sufficiently large, the final prediction remains unchanged: 𝑚𝑖★ > 2𝐿ℎ 𝐿mod 𝜀 𝑣 + 𝑐𝑖 𝑒 −Δ𝑖 , (27) where 𝑚𝑖★ is the classification margin under the ideal prompt and 𝐿ℎ is the Lipschitz constant of the frozen classifier with respect to the prompt representation. This condition indicates that the class
Dataset
Classes
Train
Test
Noise Type Objective
Caltech101 [6] Flowers102 [34] OxfordPets [36] UCF101 [39] DTD [4] EuroSAT [11]
100 102 37 101 47 10
4,128 4,093 2,944 7,639 2,820 13,500
2,465 2,463 3,669 3,783 1,692 8,100
Synthetic Synthetic Synthetic Synthetic Synthetic Synthetic
Food101N [24]
101
310,009 30,300 Real-world Food categories
5.2
Objects Flowers Pets Human actions Textures Satellite scenes
Baselines and Implementation Details
In this study, our baselines for comparison include CoOP [55], GCE [53], JoAPR [9], and NLPrompt [35], which are representative methods for robust prompt learning under noisy labels. We adopt the ResNet-50 (RN50) [10] and ViT-B/16 [21] as the image encoders. Unless otherwise clarified, we report the evaluation results with RN50 by default. For the fair comparison with baselines, we adopt the same hyperparameter settings as CoOP [55], JoAPR [9], and NLPrompt [35] with the SGD optimizer with a learning rate of 0.002 under a cosine annealing scheduler. All main experiments are conducted on an Nvidia RTX 4090 GPU and PyTorch framework [37]. We use accuracy as the main evaluation metric. Results are averaged over 3 different random seeds to ensure the fairness. Mixed precision training [33] is used to accelerate the training process. By default, the prompt length is 16, the number of attention heads is 8, no class-specific initialization is applied, and the number of shots is 16. The training batch size is 16 for all methods. We perform 200 training epochs to ensure the convergence.
Seeing is Believing: Robust Vision-Guided Cross-Modal Prompt Learning under Label Noise
Conference’17, July 2017, Washington, DC, USA
Table 2: Performance (%) over different datasets with varying noise rates. The bold denotes the best performance.
OxfordPets [36]
UCF101 [39]
Caltech101 [6]
50.0%
62.5%
75.0%
CoOp [55] GCE [53] JoAPR [9] NLPrompt [35] VisPrompt
88.93 88.80 85.57 93.87 95.57
83.50 88.33 81.23 92.57 95.29
77.93 86.73 74.60 92.73 93.99
70.10 84.07 70.23 89.90 92.37
55.60 78.37 67.90 84.77 89.77
37.17 70.37 66.93 76.80 77.73
CoOp [55] GCE [53] JoAPR [9] NLPrompt [35] VisPrompt
56.00 61.00 58.07 62.97 66.55
49.57 59.83 57.70 61.23 62.71
43.30 56.80 56.33 59.17 62.47
34.37 50.73 53.03 55.17 56.32
27.83 43.60 48.05 49.03 53.55
CoOp [55] GCE [53] JoAPR [9] NLPrompt [35] VisPrompt
76.50 82.13 75.13 81.4 82.83
69.23 78.60 61.10 76.51 80.58
61.67 74.67 60.90 75.58 76.90
52.33 63.13 63.63 65.02 67.70
CoOp [55] GCE [53] JoAPR [9] NLPrompt [35] VisPrompt
76.50 85.63 84.00 86.13 89.94
66.73 84.60 83.26 83.53 89.56
60.33 83.67 83.20 83.51 89.78
CoOp [55] GCE [53] JoAPR [9] NLPrompt [35] VisPrompt
69.03 74.00 72.83 74.83 78.72
63.40 73.63 71.17 73.40 78.88
CoOp [55] GCE [53] JoAPR [9] NLPrompt [35] VisPrompt
86.43 92.00 90.30 91.73 93.14
81.03 90.90 90.45 91.13 92.49