When Prompts Ignore Structure: Graph-Based Attribute Reasoning for Calibrated VLMs Tanay Sodha* 1 and Aditya Sharma* 1 and Ramya Hebbalaguppe2 and Vinti Agarwal1 and Pranav Murthy Yeluripaty1 1 Department of Computer Science and Information Systems, Birla Institute of Technology and Science, Pilani, India 2 TCS Research, New Delhi, India
arXiv:2607.07395v1 [cs.CV] 8 Jul 2026
Abstract Reliable confidence estimation remains a key limitation of test-time adaptation in vision– language models (VLMs), where prompt tuning improves zero-shot accuracy but often degrades calibration due to entropy-driven overconfidence. Prior approaches mitigate this using LLM-derived class attributes and contrastive regularization, yet treat attributes independently, ignoring their relational structure. We propose A RG T CA, which represents (class, attribute) pairs as nodes in a Symbolic Attribute Graph and trains a Graph Attention Network (GAT) via contrastive objectives to produce structurally informed embeddings capturing inter-attribute dependencies.
C-TPT (4.24)
TCA (2.56)
O-TPT (3.62)
A-TPT (2.67)
Ours (1.62)
Figure 1: Reliability diagrams on Caltech101 dataset (ViT-B/16, ECE(↓) in parentheses). TCA is overconfident (bars below diagonal); while TPT, C-TPT, OTPT, and A-TPT overcorrect into underconfidence (bars above diagonal). A RG T CA -D ISC (ours) achieves the closest alignment with the calibration diagonal across all confidence bins. Full results across all datasets are provided in Figure 8 in Appendix.
We introduce two attribute selection strategies: A RG T CA -D IV for intra-class diversity and A RG T CA -D ISC for inter-class discrimination. Experiments across 9 benchmarks show that A RG T CA -D IV reduces average ECE(↓) by ∼ 37% over baselines, whereas A RG T CA D ISC consistently performs as the second-best variant, reducing average ECE by ∼ 17% over baselines. These results suggest that modeling symbolic attribute interactions provides a principled approach for reliable test-time adaptation in VLMs.
1
TPT (4.59)
et al., 2022) addresses this by optimizing prompt tokens for each test image using only its own augmented views, with no labeled data. However, TPTs entropy minimization objective inherently drives the model toward overconfident predictions, producing miscalibrated outputs, quantified by the Expected Calibration Error (ECE) (Guo et al., 2017). It poses a fundamental barrier to deploying VLMs in safety-sensitive applications such as healthcare diagnostics and autonomous systems, where unreliable uncertainty estimates can have serious consequences. A key insight motivating recent work is that calibration and accuracy are largely decoupled: different prompts can achieve nearly identical top-1 accuracy while exhibiting significantly divergent ECE (Yoon et al., 2024). This implies that the geometry of the textual feature space, rather than predictive accuracy alone, governs how well-calibrated a
Introduction
Vision-language models (VLMs) such as CLIP (Radford et al., 2021) have shown strong zero-shot image recognition by aligning images and text in a shared embedding space through large-scale contrastive pretraining. CLIP performs zero-shot classification of an image by computing cosine similarities between its visual embedding and classconditioned text features generated from prompt templates such as “a photo of a {class}”. While effective, such templates are suboptimal and domainagnostic. Test-time prompt tuning (TPT) (Shu * Equal contribution.
1
VLMs predictions are. C-TPT (Yoon et al., 2024) and O-TPT(Sharifdeen et al., 2025) exploit this by adding a feature-dispersion loss and angular dispersion respectively, that spreads class-conditioned text features across the hypersphere. TCA (Hebbalaguppe et al., 2025) complements these geometric objectives by adding LLM-extracted visual attributes for semantically grounded prompt initialization, combined with a contrastive intra-class and inter-class regularization over the resulting text embeddings, achieving strong calibration.
closely aligned with the perfect calibration diagonal, outperforming existing prompt-tuning baselines. Our key contributions are: • We identify two structural failures in TCA: near-synonymous intra-class attributes that cluster on the hypersphere, and ubiquitous cross-class attributes that carry no discriminative capacity yet receive no suppression. We construct a Symbolic Attribute Graph (SAG) from class specific attributes and train a Graph Attention Network via supervised contrastive learning to produce relational embeddings capturing intra-class complementarity and cross-class redundancy.
Structural limitations of TCA. Despite strong calibration performance, TCA exhibits two structural failures that directly degrade the geometry of class-conditioned text features. First, attributes are selected as a flat set based on cosine-similarity without any relational context: on texture-rich datasets such as DTD, attributes such as rough, bumpy, and grainy are nearly collinear in CLIP’s embedding space, so top-M ′ selection clusters attribute embeddings tightly and undermines the separation that calibration requires. Second, attributes shared ubiquitously across different classes—such as mammal across leopard, dolphin, and beaver in Caltech101—carry diminished discriminative capacity, leaving their embeddings proximate across class boundaries. Both the failure modes are invisible to TCA’s selection procedure, which operates purely on similarity based ordering of LLM generated attributes without any notion of relational geometry between attributes. We propose A RG T CA (Attribute Relation Graph for Test-time Calibration Adaptation), a graphbased framework that addresses both structural failures through a Symbolic Attribute Graph (SAG) constructed by representing (class, attribute) pairs as nodes, trained offline via a Graph Attention Network with no task supervision. Two selection strategies derived from the graph — A RG T CA -D IV for intra-class diversity and A RG T CA -D ISC for interclass discrimination. Both strategies operate at zero additional test-time cost — Note: the graph is trained once offline and selection is performed before tuning begins. We validate A RG T CA across nine benchmark datasets spanning fine-grained recognition, texture classification, and domain-shifted imagery and demonstrate empirically that graph-structured relational reasoning over attributes is a key driver of calibration improvement — as shown in Figure 1, A RG T CA produces reliability diagrams most
• We introduce two selection strategies from graph-refined embeddings: A RG T CA -D IV, maximising intra-class semantic complementarity; and A RG T CA -D ISC, maximising distance from all other-class embeddings. Both strategies operate at zero additional test-time cost and are evaluated across nine benchmarks using CLIP ViT-B/16.
2
Related Work
2.1
Test-Time Adaptation of VLMs
Foundation models such as CLIP (Radford et al., 2021; Jia et al., 2021) enable zero-shot classification by aligning image and text in a shared embedding space. While learned prompt methods like CoOp (Zhou et al., 2022b) and CoCoOp (Zhou et al., 2022a) improve over hard prompts, they require labeled data, limiting zero-shot applicability. TPT (Shu et al., 2022) removes this constraint by adapting prompts via entropy minimization over augmented views of a single test image, extended further by DiffTPT (Feng et al., 2023) and PromptAlign (Mirza et al., 2023) through improved augmentation and distribution alignment. However, entropy minimization drives overconfident predictions across all TPT-family methods, significantly degrading calibration relative to the zero-shot baseline(Yoon et al., 2024). 2.2
Calibration of Vision-Language Models
Post-hoc methods such as temperature scaling (Guo et al., 2017) and Platt scaling (Platt, 1999) require labeled validation data, making them impractical at test time. C-TPT (Yoon et al., 2024) establishes that TPT degrades calibration, and that wellcalibrated prompts exhibit higher Average Text Fea2
ture Dispersion (ATFD). O-TPT (Sharifdeen et al., 2025) refines this, showing that ATFD displaces the feature centroid without ensuring pairwise angular separation — the operative quantity for calibration on the unit hypersphere. A-TPT (Ahamed et al., 2026) addresses a further degeneracy in O-TPT when N > |D| (N : #datapoints, D: feature dimensions), enforcing pairwise angular separation without dimensional constraints. TCA (Hebbalaguppe et al., 2025) initializes prompts with LLMextracted visual attributes and applies intra- and inter-class contrastive regularization, but treats attributes as a flat set ignoring relationships — a gap our work addresses. 2.3
and inter-class redundancy. These drive two principled selection strategies: A RG T CA -D IV, which selects the most semantically complementary attribute pair per class, spanning the widest semantic range for diverse prompt adaptation signals; and A RG T CA -D ISC, which scores each attribute by its mean angular distance from all other-class nodes and selects the top-M ′ per class — these maximally class-specific descriptors strengthen TCA’s inter-class regularizer. Figure 2 gives an overview of the proposed graph-informed attribute selection. In Phase 2, selected attributes initialize the standard TCA test-time tuning procedure without modification. The graph’s contribution is purely structural: deciding which attributes appear, not how they are represented. Moreover, graph training and attribute selection are performed once offline before testing, incurring no additional test-time cost.
Graph-Based VLM Adaptation
GNNs (Kipf and Welling, 2017; Veličković et al., 2018) have proven effective for learning over relational data. In VLM adaptation, GraphAdapter (Li et al., 2023) uses dual knowledge graphs for supervised few-shot transfer, HGCLIP (Zheng et al., 2024) encodes label taxonomies for hierarchical classification, and VCGPrompt (Wang et al., 2026) builds visual concept graphs for prompt learning. These works demonstrate that relational structure over the label space improves representation quality in VLMs, however are designed to operate in supervised settings, not for zero-shot calibration. In contrast, A RG T CA utilizes symbolic graph edges in a fully label-free setting, specifically targeting calibration improvement. A RG T CA occupies the intersection of all three research threads retaining TCA’s semantically grounded prompt initialization, corrects its flat-set attribute selection through a Symbolic Attribute Graph, and directly optimizes the geometric properties of text feature space that govern calibration — none of the prior works simultaneously achieves.
3
3.1
Preliminaries
Notation Let C = {c1 , . . . , cK } be a set of K class labels and Ak = {a1 , . . . , aM } be the set of M LLM-generated visual attributes for class k, giving N = K × M total (ck , aj ) pairs. A full list of notations is provided in Table 5 in Appendix. TCA background TCA (Hebbalaguppe et al., 2025) initializes soft prompts using visual attributes and defines per-class text features via attribute-conditioned prompt embeddings. For each class k and each of its M ′ selected attributes, a prompt is assembled as tk,m = p ⊕ am ⊕ ck ,
(1)
where p is the learnable soft context (initialised to "a photo of a"), and am , ck are the token embeddings of the attribute and class name strings, respectively. The text feature is fk,m = g(tk,m ), where g(·) denotes the frozen CLIP text encoder. Class probabilities combine all attribute prompts via a bag-of-attributes softmax aggregation:
Methodology
A RG T CA replaces TCA’s (Hebbalaguppe et al., 2025) similarity-based attribute selection with a graph-informed selection via two phases. In Phase 1, a Symbolic Attribute Graph (SAG) is constructed over all (class, attribute) pairs, with intra-class edges (Eintra ) encoding within-class attribute relations and inter-class edges (Einter ) encoding cross-class sharing. A Graph Attention Network (GAT) is trained with a supervised contrastive objective (Eq. 7), producing relational embeddings that capture intra-class complementarity
′
M X
p(y = k | x) =
exp fimg (x) · fk,m /τ
m=1
(2)
′
K X M X
exp fimg (x) · fk′ ,m /τ
k′ =1 m=1
where τ is CLIP’s learned logit scale. The test-time objective minimizes: LTCA = LTPT − α Linter + β Lintra 3
(3)
1 Attribute generation from LLM
2
Class- Leopard
ocean
Attributes: Feline, mammal, Carnivore, predator, fur, …
Attributes – sea, mammal, aquatic, ocean, water, …
GAT propagation
Leopard
water
GAT
mammal
Dolphin
sea
ARG-TCA_DIV: intra class diversity
Eq:§8
carnivore predator
fur
4 Select M' out of M attributes per class
feline mammal
aquatic
Class - Dolphin
3
SAG- Symbolic Attribute graph
ARG-TCA_DISC: inter class discrimination
§6
mammal
Eq:§9 fur
water
5
Beaver
Class - Beaver Attributes: rodent, water, mammal, construct, fur, …
construct
rodent
Selected attributes
§7
Eq: §4
Figure 2: Overview of A RG T CA attribute selection pipeline. ❶ An LLM generates M candidate attributes per class. ❷ These are used to construct the Symbolic Attribute Graph (SAG) with intra-class (Eintra ) and inter-class (Einter ) edges. ❸ A Graph Attention Network (GAT) is trained via supervised contrastive loss (LSupCon ) to propagate structural relationships across the SAG, producing attribute embeddings. ❹ Two selection criteria — A RG T CA -D IV (intra-class diversity) and A RG T CA -D ISC (inter-class discrimination) — select the most informative M ′ attributes ′ per class. ❺ The final selected attributes form the matrix A ∈ RM ×K , used for downstream prompt tuning.
with α weights how aggressively class centroids µk are pushed away from the global centroid µ, and β weights how tightly each class’s attribute embeddings fk,m are pulled toward their class centroid µk . LTPT is the entropy of the average prediction over the ⌊B · psel ⌋ most confident of B augmented 1 P views. Linter = K k ∥µk − µ∥2 is Pthe mean distance of class centroids µk = M1 ′ m fk,m from the global centroid µ;P maximizing it spreads classes 1 1 P apart. Lintra = K k M ′ m ∥fk,m − µk ∥2 is minimized to make each class’s attribute prompts consistent. Only p is updated; all other parameters are frozen. 3.2 3.2.1
3.2.2 Symbolic Attribute Graph (S AG) We define the edge set E via a symbolic rule over the node vocabulary: n o E = (i, j) i ̸= j and ci = cj or ai = aj (5) | {z } | {z } Eintra
where both directions are included and self-loops are omitted. In equation 5 Eintra connects all attribute nodes belonging to the same class (ci = cj ), enabling the GAT to jointly aggregate every attribute perspective of a class; Einter connects nodes that sharing a common attribute token across different classes — for instance, mammal links leopard, dolphin, and beaver in Caltech101 — allowing the GAT to propagate how the same descriptor is contextualized differently across class neighborhoods. For K classes each with M attributes, |Eintra | = K · M · (M − 1) is guaranteed regardless of the data distribution.
Phase 1 — Symbolic Attribute Graph and GAT Training Node Initialization
We adopt the M visual attributes per class provided by Hebbalaguppe et al. (2025), generated offline via GPT-4 prompting across all nine benchmark datasets. Specifically, for each class ck , an LLM is prompted with the class name to produce a list of descriptive visual attributes am , which are subsequently ranked in descending order of relevance by cosine similarity between the attribute and class name embeddings (Hebbalaguppe et al., 2025). Each class-attribute pair (ck , aj ) is represented as a graph node, initialized with ℓ2 -normalized EOS hidden state obtained by encoding the phrase “a aj of a ck ” through frozen CLIP text encoder,g(·): (0)
hi
= g("a aj of a ck ") ∈ Rd
Einter
3.2.3 GAT & hybrid Objective We use GAT to propagate feature information across Eintra and Einter edges so that each attribute node aggregates both the complementary perspectives of its own class and the cross-class context of any shared attribute, producing node embeddings h(L) where same-class attributes are pulled into a compact, coherent cluster while cross-class attribute representations are pushed apart. Concretely, at layer l, the updated embedding of node i is computed as: X H (6) hli = ELU αjh · Wh hl−1 j h=1
(4) 4
j∈N (i)
where ∥ denotes head-wise concatenation over H attention heads, N (i) is the set of neighbors of (l) node i under E, and Wh ∈ Rd/H×d is the headspecific linear projection at layer l. The attention (l,h) coefficient αji measures the relative importance of neighbor j to node i under head h, and ELU induces non-linearity. Full architectural details including the attention coefficient computation are given in section B of appendix. GAT Training Objective The GAT is trained using supervised contrastive loss (Khosla et al., 2020) where same-class (ck , aj ) nodes form positives, and all inter-class nodes form negatives.
token embeddings. The per-class text feature is fk,m = g(tk,m ) and class probabilities follow Eq. 2. The full positional breakdown is given in Table 4. prompt tk,m = [SOS] | {z } pos 0
i
j̸=i
2 and A = nodes that have at least one positive. This trains the GAT to produce class-cohesive representations in h(L) space while discriminating against all other classes.
Attribute Selection (Post training)
After GAT training, the learned h(L) is used to select M ′ of the M attributes per class using one of two criteria. Both strategies operate entirely offline at zero test-time cost.
(L)
i̸=j∈Ik
(L)
· h̃j .
Experimental Setup
4.1
Datasets and Baselines
• C-TPT (Yoon et al., 2024): Adds a text feature dispersion loss (ATFD) to TPT (Shu et al., 2022) to improve calibration.
(8) • O-TPT (Sharifdeen et al., 2025): Replaces ATFD with pairwise angular separation between class centroids.
For each node aki in class ck ,
A RG T CA -D ISC: we compute discriminability score measuring its mean angular distance from all other-class nodes: score(aki ) =
4
• TCA (Hebbalaguppe et al., 2025): The direct predecessor, which initializes prompts with LLM-generated visual attributes and applies contrastive intra- and inter-class regularization. Our method builds on and improves upon TCA’s attribute selection procedure.
A RG T CA -D IV — Intra-class diversity For each class ck ∈ C, select the pair (aki , akj ) with the minimum cosine similarity in the graph-refined embedding space — i.e., the most semantically complementary pair: (aki , akj ) = arg min h̃i
pos nctx +1 ··· 76
We evaluate on nine benchmarks datasets, using ViT-B/16 as CLIP backbone and following the evaluation protocol of baselines (He et al., 2016; Radford et al., 2021). Dataset details are provided in section A in appendix. Furthermore, we compare our proposed approach against recent, state-of-theart approaches:
(7)
where h̃(L) = h(L) /∥h(L) ∥
3.3
pos 1-nctx
akm , cls tokens, . . . (10) {z } |
Test-time objective & inference For each test image, the top ⌊B · psel ⌋ most confident of B augmented views are selected and a single AdamW step minimizes LTCA (Eq. 3) on p. Final prediction uses the non-augmented image with the updated p, which is reset before each new test image. Figure 7 in appendix illustrates the prompt-tuning process.
exp h̃⊤ i · h̃p /τ 1 X −1 X LSupCon = log X |A| i∈A |P (i)| exp h̃⊤ · h̃j /τ p∈P (i)
p |{z}
• A-TPT (Ahamed et al., 2026): Fixes O-TPT’s degeneracy when classes exceed embedding dimension.
1 X (L) (L) 1 − cos h̃i , h̃j , |Ω̄k | j ∈Ω / k
(9) where Ωk denotes the global node indices for class k and Ω̄k all other-class indices. The top M ′ attributes by score are selected per class.
All baselines are re-evaluated under identical hyperparameters tuned on the Caltech101 validation set, following the protocol of TCA (Hebbalaguppe et al. (2025)).
3.4
4.2
Phase 2 — Test-Time Tuning
Prompt construction For each ck and its M ′ attributes selected in Phase 1, prompts are assembled via Eq. 1, with am and ck as frozen CLIP
Implementation Details
All experiments are done on a single A100 80GB GPU. The CLIP backbone is fully frozen throughout both phases. 5
ECE Comparison (lower is better) Caltech101
Phase 1 — Offline Graph Training. Each dataset’s attribute graph is constructed from Nattr = 10 LLM-generated attributes per class (from Hebbalaguppe et al. (2025)), giving K×10 graph nodes. The Graph Attention Network (AttributeGAT; 2 layers, 4 heads, 512-dimensional node features) is trained for 100 epochs with supervised contrastive loss (Eq. 7), temperature τ = 0.07, and the Adam optimizer with learning rate 10−3 . Phase-1 training is done offline and completes in under two minutes per dataset.
3.9
Pets 3.9
5.0
2.9
2.6 2.4
Cars 3.1
2.8
1.9 2.4
2.1
4.1 7.7
2.6
11.2
3.0
Phase 2 — Test-Time Prompt Tuning. We follow the standard TCA evaluation protocol: soft context initialised to “a photo of a” (nctx = 4), 64 augmented views per test image with the top 10% selected by entropy, and a single gradient step per image with batch size 64. Regularization weights: ViT-B/16 uses α = 10, β = 35; M ′ =2 attributes per class are selected from the graph-refined embeddings via either the diversity (A RG T CA -D IV) or discriminability (A RG T CA -D ISC) criterion.
Results
5.1
Main Results
10.2
8.2
TCA
O-TPT
17.4
13.9
20.0
17.7
22.6
EuroSAT
21.4
10.4
DTD
12.6 14.7
Food101
Aircraft A-TPT
ARGTCA-DIV
Figure 3: Radar plot comparing ECE (lower is better, smaller area is better) of TCA, O-TPT, A-TPT, and A RG T CA -D IV (Ours) across nine datasets (ViT-B/16). A RG T CA -D IV consistently occupies the innermost region, with the largest gains on Aircraft, Flower, and EuroSAT.
(red) predictions at high confidence levels, indicating systematic overconfidence. A RG T CA -D ISC achieves the lowest ECE of 1.62% at 93.2% accuracy, with incorrect predictions concentrated at low confidence and correct predictions (green) dominating the high-confidence region. By contrast, O-TPT (ECE: 3.61%) and TCA (ECE: 2.56%) exhibit red-green mixing in mid-confidence bins (2– 4), while A-TPT (ECE: 2.67%) partially improves but still misclassifies samples at low confidence. The radar plot (Figure 3) confirms this trend across all nine benchmarks: A RG T CA consistently occupies the innermost region, with gains most pronounced on Aircraft (ECE: 1.21% vs. 8.19% for A-TPT), Food101 (1.75% vs. 2.02% for A-TPT), and Flower (2.88% vs. 10.38% for A-TPT). Reliability diagrams (Figures 1, 8) further corroborate this, with A RG T CA showing the closest alignment to the calibration diagonal — particularly in highconfidence bins (0.8–1.0) — confirming that graphbased attribute selection suppresses overconfidence without inducing underconfidence.
Table 1 shows that A RG T CA -D IV achieves the best average ECE of 4.45%, reducing miscalibration by 26.7%, 31.5%, 42.4%, and 48.0% over C-TPT, TCA, O-TPT, and A-TPT respectively, while improving average accuracy simultaneously. A RG T CA -D ISC achieves the highest average accuracy of 64.68%, with an ECE of 5.90% — a 2.8%, 9.2%, 23.7%, and 31.0% reduction over C-TPT, TCA, O-TPT, and A-TPT respectively. Notably, A-TPT achieves competitive accuracy (64.84%) but at a significantly higher calibration cost (ECE: 8.55%). A RG T CA -D ISC breaks this trade-off — matching A-TPT’s accuracy within 0.2% while reducing its ECE by 31%, confirming that graphstructured attribute reasoning provides consistent calibration gains. 5.2
6.4 1.8 6.0
3.4
Metrics We report top-1 accuracy (Acc, %, ↑) and Expected Calibration Error (ECE, %, ↓) computed with 20 equal-width bins following Hebbalaguppe et al. (2025). Lower ECE indicates bettercalibrated predictions.
5
14.8
12.2
2.2
9.4
4.2
2.2 3.4
1.7
5.9
Flower
5.8
3.0
3.4
UCF101
6.6
3.5
5.3
Accuracy–Calibration Trade-off
The two selection strategies encode complementary objectives. A RG T CA -D IV selects the most semantically complementary attribute pair per class, maximizing angular spread in h(L) space. The resulting diverse prompt initialization provides strong gradi-
Calibration Quality: Confidence Distributions and Reliability
The confidence bin plots (Figure 4) reveal that TCA, O-TPT, and A-TPT accumulate incorrect 6
Aircraft
DTD
EuroSAT
UCF101
Average
Food101
Flower
Cars
Pets
Caltech
Metric
Method
23.61 16.48 23.85 4.38 24.96 4.52 23.46 7.75 23.34 13.64 24.69 4.83 25.05 4.25
46.70 21.35 46.04 12.27 44.73 11.26 46.87 19.75 47.22 19.81 47.34 3.17 46.63 6.36
42.79 21.49 27.8 15.16 45.52 11.35 42.85 20.92 43.29 17.53 32.84 15.83 39.53 19.81
67.27 13.01 59.7 3.89 66.9 5.25 66.90 3.56 67.64 6.11 69.63 3.05 68.70 3.81
64.62 11.70 61.49 6.07 63.31 6.50 64.60 7.73 64.84 8.55 63.99 4.45 64.68 5.90
Pre-trained Backbone: CLIP Vit-B/16 | Embedding dimension: 1024-d TPT (NeurIPS’22) C-TPT (ICLR’24) TCA (ECML’25) O-TPT (CVPR’25) A-TPT (ICLR’26) A RG T CA -D IV (Ours) A RG T CA -D ISC (ours)
Acc. ECE Acc. ECE Acc. ECE Acc. ECE Acc. ECE Acc. ECE Acc. ECE
93.83 4.59 93.35 4.24 93.82 2.56 93.67 3.62 93.75 2.67 92.82 2.52 93.19 1.62
87.08 5.77 84.14 2.77 90.51 6.3 88.55 1.92 88.49 2.23 89.48 3.44 88.91 3.37
66.32 5.25 65.45 1.94 65.92 7.85 65.65 1.80 66.44 2.54 65.07 2.75 66.47 4.29
69.31 13.27 69.79 5.21 69.18 3.67 69.96 7.15 69.83 10.38 70.36 2.74 69.59 3.57
84.70 4.05 83.28 3.77 69.18 5.28 83.52 3.13 83.57 2.20 83.64 1.75 83.85 2.06
Table 1: Top-1 accuracy (Acc.%, ↑) and Expected Calibration Error (ECE, %, ↓) on nine benchmarks on ViT-B/16. red: best per dataset and metric; orange: second best. ECE Sensitivity: sweep ( = 35 fixed)
ent signals for both Lintra and Linter , driving higher accuracy alongside well-calibrated text distributions. A RG T CA -D ISC instead targets attributes maximally distant from other-class embeddings, initializing prompts with high inter-class separation and more conservative updates — effective on datasets with severe inter-class confusion (e.g., Caltech101, Aircraft) but less consistent across all nine benchmarks.
ARGTCA-DIV ECE ( )
9 8
8 7
ECE (%)
7
ECE (%)
ARGTCA-DISC ECE ( )
Caltech101 OxfordPets DTD = 10 (default)
6 5 4
4
2 5
10
20
40
5
10
20
40
Figure 5: ECE vs. α (inter-class regularization weight) on Caltech101, OxfordPets, and DTD (β=35 fixed). ECE Sensitivity: sweep ( = 10 fixed)
ARGTCA-DIV ECE ( )
6.0
4.5
8 7
ECE (%)
ECE (%)
ARGTCA-DISC ECE ( ) 9
Caltech101 OxfordPets DTD = 35 (default)
5.5 5.0
4.0
6 5
3.5
4
3.0
3 2
2.5 15
25
35
50
15
25
35
50
Figure 6: ECE vs. β (intra-class regularization weight) on Caltech101, OxfordPets, and DTD (α=10 fixed).
(b) O-TPT
5.4 (c) A-TPT
5
3
3
Recommendation: A RG T CA -D IV for broad crossdataset calibration; A RG T CA -D ISC when interclass confusion dominates.
(a) TCA
6
Ablation Studies
All ablations use ViT-B/16 on Caltech101, OxfordPets, and DTD. We ablate three components of A RG T CA: (i) edge structure — S AG trained with E=∅, removing message passing while retaining contrastive training and Div/Disc selection; (ii) GAT training — Div/Disc applied directly to raw CLIP EOS embeddings h(0) , skipping Phase 1 entirely; and (iii) selection crite-
(d) A RG T CA (Ours)
Figure 4: Confidence bin plots on Caltech101 (ViTB/16). Green/red indicate correct/incorrect predictions. A RG T CA -D ISC achieves the lowest ECE with the cleanest separation between correct and incorrect predictions across confidence bins.
7
Variant
Sel.
Div Disc Div (i) Zero edges Disc Div (ii) Raw EOS Disc Div (iii) Random sel. Disc A RG T CA (Full)
Caltech101
OxfordPets
Acc
ECE
Acc
ECE
Acc
ECE
92.82 93.19 93.18 93.51 92.21 93.39 93.10 92.25
2.52 1.62 2.36 1.74 2.73 2.60 2.53 2.43
89.48 88.91 89.32 90.00 88.74 88.99 88.63 88.50
3.44 3.37 3.84 3.50 3.87 3.71 3.44 3.55
47.34 46.63 45.04 45.74 46.81 45.98 45.92 44.27
3.17 6.36 4.91 5.31 5.65 8.95 5.16 6.33
Attributes selected by TCA vs. A RG T CA. TCA selects top-M ′ attributes by cosine similarity to the class name, surfacing near-synonymous descriptors that cluster tightly on the hypersphere. A RG T CA D IV selects the maximally complementary pair in h(L) space; A RG T CA -D ISC selects the pair most distant from all other-class embeddings. Per-class examples are in Table 3.
DTD
Table 2: Ablation study on Caltech101, OxfordPets, and DTD (ViT-B/16). Acc (%, ↑) / ECE (%, ↓). red: best per column; orange: second best. Div = A RG T CA -D IV; Disc = A RG T CA -D ISC.
Hyperparameter sensitivity (α, β). Both weights are stable across α ∈ {5, 10, 20, 40} and β ∈ {15, 25, 35, 50}: ECE varies by at most ∼2% across the full sweep, with the ViT-B/16 defaults (α=10, β=35) lying in the stable plateau (see Figures 5–6). Full ECE sweep for both selection strategies are provided in Appendix C.
rion — Phase 1 run in full but with random attribute selection (M ′ =2, seed 42). We additionally sweep α ∈ {5, 10, 20, 40} (β=35 fixed) and β ∈ {15, 25, 35, 50} (α=10 fixed). A detailed per-ablation description and hyperparameter sweep figures are in shown Appendix C. The ablation results (Table 2) confirm the contribution of each component. Removing edge structure (i) degrades ECE on DTD for both Div (3.17→4.91) and Disc (6.36→4.91). Skipping GAT training entirely (ii) causes the largest degradation: DTD ECE rises to 5.65 (Div) and 8.95 (Disc), confirming that relational contrastive training is critical for texture-heavy datasets. Random selection (iii) degrades DTD Div ECE from 3.17 to 5.16 with minimal accuracy impact, directly isolating the geometric Div/Disc criteria as the operative calibration mechanism. Dataset Class
TCA (top-2)
A RG T CA -D IV
A RG T CA -D ISC
Caltech101 airplane
aircraft, travel
jet, airline
image, digital design, chair images, human motorbike, pillion
transportation, altitude digital, vintage design, director’s natural, adults seat, suspension
Abyssinian, cats Dog, Boxer Feline, Cat Adorable, Persian Shiba, Dog
restful, observant Athletic, Friendly Tabby, Fluffy Adorable, Lying Cute, Foxy
cats, feline Boxer, Pet Kitten, Coon Persian, Adorable Inu, Shiba
banded bumpy crystalline
design, casual uneven, unequal natural, crystals
material, colorful unequal, geometric natural, glittering
dotted woven
variation, design variation, fashion interwoven, crafts- material, organic manship
casual, colorful knobby, indented shimmering, crystals dots, stylized handwoven, craftsmanship
camera chair face motorbike OxfordPets Abyssinian Boxer Maine Coon Persian Shiba Inu
6
Conclusion
We present A RG T CA, a test-time calibration framework for CLIP that replaces flat attribute selection in TCA with graph-relational reasoning. A GAT trained via supervised contrastive loss over the Symbolic Attribute Graph (S AG) produces attribute embeddings encoding intra-class diversity and inter-class discriminability, giving rise to two selection criteria: A RG T CA -D IV and A RG T CA D ISC. Across nine benchmarks, A RG T CA -D IV achieves the lowest average ECE (4.45 ), reducing miscalibration by up to 48%. Ablations confirm that edge topology, contrastive training, and geometric selection each contribute independently. Future works include a unified selection objective jointly optimiing diversity and discriminability, and cross-dataset graph transfer to remove the perdataset offline build requirement.
dslr, digital elegant, chair emotion, casual suspension, motorbike
Limitations LLM attribute quality. The method inherits TCA’s dependence on LLM-generated attribute quality. Domains where LLMs produce generic or noisy attributes (e.g., satellite imagery) remain challenging. Richer, domain-specific attribute generation strategies may be needed for such cases.
DTD
Accuracy–calibration tradeoff. A RG T CA -D IV and A RG T CA -D ISC are complementary but not jointly optimal: across benchmarks, A RG T CA -D IV tends to improve accuracy while A RG T CA -D ISC tends to reduce ECE, and no single variant dominates on both metrics simultaneously. Practitioners must choose a selection strategy based on deployment priority, which limits plug-and-play usability
Table 3: Representative attribute selection per class across three datasets. (A RG T CA -D IV, A RG T CA -D ISC) avoid near-synonymous descriptors that cluster tightly in CLIP embedding space, unlike TCA.
8
without prior knowledge of the target domain.
References
Dataset-specific graph. The S AG and GAT are trained independently per dataset, so applying A RG T CA to a new benchmark requires re-running Phase 1. The learned graph structure does not transfer across datasets or domains, and Phase 1 cannot be run when the class vocabulary is unavailable at build time, constraining applicability in fully zero-shot settings.
Shihab Aaqil Ahamed, Udaya S. K. P. Miriya Thanthrige, Ranga Rodrigo, and Muhammad Haris Khan. 2026. A-tpt: Angular diversity calibration properties for test-time prompt tuning of visionlanguage models. Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. 2014. Food-101 – mining discriminative components with random forests. In Proceedings of the 13th European Conference on Computer Vision, ECCV, pages 446–461. Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. 2014. Describing textures in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3606–3613. Li Fei-Fei, Rob Fergus, and Pietro Perona. 2004. Learning generative visual models from few training examples. In Proceedings of the CVPR 2004 Workshop on Generative-Model Based Vision. Chun-Mei Feng, Kai Yu, Yong Liu, Salman Khan, and Wangmeng Zhong. 2023. Diverse data augmentation with diffusions for effective test-time prompt tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, ICCV, pages 2704–2714. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, ICML, pages 1321– 1330. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 770–778. Ramya Hebbalaguppe, Tamoghno Kandar, Abhinav Nagpal, and Chetan Arora. 2025. Prompting without panic: Attribute-aware, zero-shot, test-time calibration. arXiv preprint arXiv:2506.22819. Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. 2019. EuroSAT: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217–2226. Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML. Prannay Khosla, Yonglong Tian, Huiwen Wang, Chen Liu, Jack Valmadre, Chen Tian, Mohammad Norouzi, and 1 others. 2020. Supervised contrastive learning.
9
In Advances in Neural Information Processing Systems, volume 33 of NeurIPS, pages 18661–18673.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. 2012. UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402.
Thomas N. Kipf and Max Welling. 2017. Semisupervised classification with graph convolutional networks. In International Conference on Learning Representations, ICLR.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph attention networks. In International Conference on Learning Representations, ICLR.
Jonathan Krause, Michael Stark, Jia Deng, and Li FeiFei. 2013. 3D object representations for fine-grained categorization. In ICCV Workshop on 3D Representation and Recognition.
Mengjia Wang, Fang Liu, Licheng Jiao, Shuo Li, Lingling Li, Puhua Chen, Xu Liu, and Wenping Ma. 2026. VCGPrompt: Visual Concept Graph-Aware Prompt Learning for Vision-Language Models. Pattern Recognition, 170:112012.
Xin Li, Wenxuan Lian, Zhixin Lu, Jingwen Bao, Jiaming Liu, Junliang Li, Jiaxin Lan, and Lingling Zhu. 2023. GraphAdapter: Tuning vision-language models with dual knowledge graph. In Advances in Neural Information Processing Systems, NeurIPS.
Jihoon Yoon, Jae-hun Oh, Byungho Kim, In So Kweon, and Jongwoo Choi. 2024. C-TPT: Calibrated testtime prompt tuning for vision-language models via text feature dispersion. In International Conference on Learning Representations, ICLR.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew B. Blaschko, and Andrea Vedaldi. 2013. Finegrained visual classification of aircraft. CoRR, abs/1306.5151.
Xiaobao Zheng, Liqiang Ji, Changhao Hong, Nong Ying, Mengqi Liu, and Zheng Zhang. 2024. HGCLIP: Exploring vision-language models with graph representations for hierarchical understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR.
Muhammad Uzair Mirza, Fahad Shahbaz Khalid, Fawad Siddiqui, and Salman Khan. 2023. Lafter: Labelfree test-time adaptation with language feedback for robust visual prompt tuning. In Advances in Neural Information Processing Systems, NeurIPS. Maria-Elena Nilsback and Andrew Zisserman. 2008. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022a. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, pages 16816–16825.
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. 2012. Cats and dogs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3498–3505.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022b. Learning to prompt for visionlanguage models. International Journal of Computer Vision, 130:2337–2348.
John C. Platt. 1999. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Advances in Large Margin Classifiers, pages 61–74. MIT Press. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML, pages 8748–8763. PMLR. A. Sharifdeen and 1 others. 2025. O-TPT: Orthogonality-regularized test-time prompt tuning for zero-shot generalization in vision-language models. In International Conference on Learning Representations, ICLR. Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. 2022. Test-time prompt tuning for zero-shot generalization in vision-language models. In Advances in Neural Information Processing Systems, volume 35 of NeurIPS, pages 14274–14289.
10
Appendix Contents Section
Page
A Datasets
12
B GAT Architecture Details
12
Attention and Aggregation
12
Residual, Normalisation, and Projection
12
Node Attention Score
12
C Ablation Studies
12
C.1 Attribute Comparison: TCA vs. A RG T CA -D IV vs. A RG T CA -D ISC
12
C.2 Zero Graph Edges
12
C.3 No GAT Training (Raw CLIP EOS Embeddings)
12
C.4 Random Attribute Selection
12
C.5 Hyperparameter Sensitivity (α, β)
12
D Prompt Token Layout
13
E Notation Reference
14
Figures TCA Pipeline Overview
13
Reliability Diagrams Across All Datasets (ViT-B/16)
15
Tables Attribute Comparison: TCA vs. A RG T CA -D IV vs. A RG T CA -D ISC
8
Ablation Results
8
Prompt Token Layout — A RG T CA
13
Notation Reference
14
11
A
Datasets
A two-layer MLP projects to the final embedding: z = W2 (GELU(W1 h(L) + b1 )).
We evaluate on nine benchmarks: Caltech101 (FeiFei et al., 2004) (general objects), StanfordCars (Krause et al., 2013), FGVC-Aircraft (Maji et al., 2013), OxfordPets (Parkhi et al., 2012), and Flowers102 (Nilsback and Zisserman, 2008) (fine-grained recognition), Food101 (Bossard et al., 2014) (food categories), DTD (Cimpoi et al., 2014) (perceptual textures), UCF101 (Soomro et al., 2012) (action recognition), and EuroSAT (Helber et al., 2019) (satellite land-use). We follow the standard (Zhou et al., 2022b) test splits used in TCA (Hebbalaguppe et al., 2025), evaluating on the test set only with no labeled data.
Node attention score. A scalar importance score per node, used for attribute selection in A RG T CA D IV and A RG T CA -D ISC, is computed as the mean attention coefficient across heads and layers: attn[v] =
l=1 h=1
A full list of all the notations used in the document are provided in Table 5.
Our graph-based attribute selection follows the Graph Attention Network (GAT) framework of Veličković et al. (2018), which assigns learned importance weights to neighborhood edges during message passing, enabling the model to selectively aggregate information from the most relevant neighbors. Our S AG is an undirected graph; each undirected edge {u, v} ∈ E is represented as two directed edges (u → v) and (v → u) during message passing, following standard practice in GAT implementations.
h
(11)
e = h(l) Wlayer is the linearly projected emwhere h bedding. Node representations are updated as: h
= ELUconcatH h=1
X
C.1
Attribute comparison: TCA vs. A RG T CA -D IV vs. A RG T CA -D ISC.
Zero graph edges.
To isolate the contribution of the S AG edge structure, we train the GAT with E = ∅: all nodes are present and trained via supervised contrastive loss, but no message passing occurs. Without edges, each GAT layer reduces to an independent pernode linear transform; no cross-class or withinclass structural context propagates. Attribute selection uses the same Div/Disc criteria on these edge-free embeddings. The calibration drop on DTD relative to the full model (Table 2) confirms that the S AG’s edge topology — not the contrastive objective alone — drives selection quality.
e e exp LeakyReLU(a⊤ h [hu,h ∥hv,h ]) , αuv,h = P e u′ ,h ∥h e v,h ]) exp LeakyReLU(a⊤ [h ′
Ablation Studies
C.2
Attention and aggregation. For each edge {u, v} and head h, the normalized attention coefficient is:
e (l)
C
The table 3 lists the two attributes selected per method for a representative subset of classes from Caltech101, OxfordPets, and DTD. TCA ranks attributes by cosine similarity between each attribute’s frozen CLIP embedding and the class name embedding, selecting the top-M ′ ; A RG T CA D IV selects the most complementary pair by minimum pairwise cosine similarity in h(L) space; A RG T CA -D ISC selects the pair with highest mean cosine distance to all other-class embeddings in h(L) space.
GAT Architecture Details
u ∈N (v)
u∈N (v)
(14) This scalar serves as the basis for attribute selection in both A RG T CA -D IV and A RG T CA -D ISC (Section 3.2).
Notation Reference
B
L H X 1 XX 1 αuv,h . LH |N (v)|
e u,h . drop(αuv,h ) · h
C.3
u∈N (v)
(12)
Residual, normalisation, and projection. After L layers, a residual connection (with Wres initialised to I) mitigates over-smoothing, followed by ℓ2 normalisation: e (L) + Wres h(0) . (13) h(L) = normalize h
No GAT training (raw CLIP EOS embeddings).
We skip Phase 1 entirely and apply the Div/Disc selection criteria directly to the initial embeddings h(0) (frozen CLIP EOS hidden states, Eq. 4), without any contrastive training. This tests whether GAT training adds value beyond the semantic geometry already present in CLIP’s encoder. The 12
Position 0 1–nctx nctx + 1 nctx + 2 – L−1
Content
Notes
SOS token embedding Soft prompt p ∈ Rnctx ×Dt akm Class tokens + EOS + padding
Frozen Learnable (tuned at test time) graph selected attribute Frozen
Table 4: Full positional breakdown of a prompt in A RG T CA.
Dog
Backpropagation
Training/Tuning
Learnable parameters
Frozen
Dolphin
Cat
Text encoder
Image encoder
Loss
Augmentations
Backpropagation
Figure 7: Test time prompt calibration
optimal region is α ∈ [10, 20] and β ∈ [25, 35]; the ViT-B/16 defaults (α=10, β=35) lie within this plateau, confirming robustness to precise hyperparameter choice.
DTD ECE degrades substantially (Disc: 8.95 vs. 6.36 for the full model), showing that contrastive relational training is necessary for reliable attribute discrimination on texture-heavy datasets. C.4
Random attribute selection.
We run Phase 1 in full (SAG construction, GAT training) but replace Div/Disc selection with a random draw of M ′ = 2 attributes per class (seed 42). This isolates the selection criterion: random selection degrades ECE on DTD (Div: 5.16 vs. 3.17 for the full model) while leaving Caltech accuracy largely unchanged, confirming that the geometric Div/Disc criteria are the operative mechanism for calibration improvement, not merely the graph training. C.5
Hyperparameter sensitivity (α, β).
Figures 5 and 6 show ECE as α and β are swept independently (α ∈ {5, 10, 20, 40} with β=35 fixed; β ∈ {15, 25, 35, 50} with α=10 fixed), evaluated on Caltech101, OxfordPets, and DTD for both A RG T CA -D IV and A RG T CA -D ISC. Both metrics are stable across the swept range: ECE varies by at most ∼1.5% on Caltech101 and ∼2.0% on DTD for α, and by at most ∼1.8% on DTD for β. The 13
Symbol
Shape / Type
Meaning
K
scalar
Number of classes
M
scalar
Attributes loaded per class (num_attributes= 10)
M′
scalar
Attributes selected for injection (prompt_attributes= 2)
N =K ×M
scalar
Total graph nodes
d = 512
scalar
CLIP token embedding dimension
ck
string
Class name for class k
aj
string
j-th attribute string
(ck , aj )
tuple
Node i in the graph
(0) hi (L) hi
Rd
Initial node embedding (CLIP EOS hidden state)
Rd
GAT output (refined node embedding)
p
Rnctx ×d
Soft prompt context (trainable, Phase 2)
g(·)
Rd
CLIP text encoder EOS hidden state
f (·)
Rdout
CLIP text encoder full output (after projection T)
τ
scalar
InfoNCE temperature
α, β
scalars
TCA inter / intra loss weights
Table 5: Notation used throughout this document.
14
TPT
TCA
C-TPT
A-TPT
O-TPT
A RG T CA (Ours)
DTD
Stanford Cars
Food101
Oxford Pets
EuroSAT
Aircraft
Flower102
Figure 8: Reliability diagrams across all datasets using ViT-B/16. Each row is a dataset; each column is a method. Bars above/below the diagonal (pink) indicate under/overconfidence. A RG T CA consistently shows better alignment with the diagonal across all datasets.
15
ECE Comparison (lower is better) Caltech101 3.9
Pets 3.9
5.2
2.9
2.4 2.4
Cars 4.6
4.0
1.9 3.3
2.7
4.5
1.9 3.7
4.7 7.9
8.6 1.8 5.8
2.2
9.6
2.6
11.2
3.0
14.8
12.2
2.0
6.3
Flower
5.9
2.9
3.4
11.8
8.1
15.0
20.0
18.2
22.6
EuroSAT
21.4
DTD
12.5 14.7
Food101 O-TPT
17.4
10.3
3.4
TCA
UCF101
6.6
3.4
Aircraft A-TPT
ARGTCA-DISC
Figure 9: Radar plot comparing ECE (lower is better, smaller area is better) of TCA, O-TPT, A-TPT, and A RG T CA -D ISC (Ours) across nine datasets (ViT-B/16).
16