Bridging Domain Expertise and Generalization for Performance Estimation Shuxuan Li1 Zhilin Zhao1,2,3 * Quyu Kong4 Wei-Shi Zheng1,2,3 1 School of Computer Science and Engineering, Sun Yat-sen University, China 2 Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China 3 Shenzhen Loop Area Institute, China 4 Alibaba Cloud
arXiv:2606.06335v1 [cs.LG] 4 Jun 2026
[email protected], [email protected] [email protected], [email protected]
Abstract Performance estimation under distribution shift aims to predict how a model behaves on an unlabeled test set whose distribution differs from the training data, a scenario that requires reliable indicators that can faithfully reflect model behavior without ground-truth labels. Existing approaches rely solely on the outputs of the given model whose biases are amplified once the distribution shifts, weakening the correlation with the true performance. Motivated by this limitation, we propose Fused Reference Alignment Prediction (FRAP), which leverages the complementary strengths of an external foundation model and the base model to construct a more reliable surrogate of the ground-truth labels. FRAP aligns the prediction distribution of the foundation model with that of the base model by applying temperature-scaled calibration that minimizes their divergence. The aligned predictions are fused through confidence-based weighting into a refined reference distribution that integrates robustness from the foundation model and domain-specific expertise from the base model, and performance estimation is obtained by measuring how closely the base model predictions agree with this reference. Extensive experiments across diverse datasets and architectures show that FRAP provides consistent and substantial improvements over representative performance-estimation methods under distribution shift.
1. Introduction In the conventional machine learning paradigm, models are trained on a labeled source dataset and evaluated on an unlabeled target dataset. This process implicitly relies on the i.i.d. assumption, which requires both datasets to originate from the same underlying distribution. However, real-world deployment frequently violates this assumption, resulting in significant performance degradation [21, 40]. Consequently, * Corresponding author. Code is available at https://github. com/NuyoahNasuS/FRAP
estimating model performance on unlabeled data under distribution shift is essential for building safe and reliable machine learning systems. In the absence of target labels, performance estimation requires an indicator that reliably reflects the true behavior of the model. Prior work explores a variety of heuristics, including prediction agreement [1, 19, 43, 61], confidencebased statistics [6, 10, 11, 36], and prediction-distribution characterization [31, 58]. Despite methodological differences, these approaches all depend solely on outputs produced by the model itself. Under distribution shift, however, this dependency becomes problematic, e.g., confidence no longer reliably correlates with accuracy, and agreementbased scores may increase even when predictions are consistently incorrect [12, 19, 42, 48]. These failures stem from limited generalization of the model and the biased predictions that emerge when the input distribution changes. This situation highlights the need for an external source of knowledge that can provide validation beyond the potentially biased outputs of the model under analysis. A natural solution is to incorporate an external generalized reference that remains independent of the model under analysis and supplies broader knowledge that extends far beyond the source domain. Foundation models naturally satisfy this requirement because training on large-scale diverse datasets enables strong cross-domain generalization. At the same time, the base model retains domain-specific expertise derived from task-oriented training, although its generalization ability diminishes once the input distribution shifts. These complementary properties indicate that integrating a broadly trained foundation model with a task-specialized base model can provide a more stable and informative reference signal for performance estimation. Achieving this integration is nontrivial because the two prediction distributions arise from different training paradigms and feature spaces, and models frequently exhibit notable miscalibration under distribution shift. These challenges motivate a principled mechanism that establishes a consistent probabilistic space for both models and combines their predictions in a
reliability-aware manner. Building on this insight, we propose the Fused Reference Alignment Prediction (FRAP) framework, which unifies strengths from both the foundation model and the base model. FRAP adaptively aligns the prediction distribution generated by the foundation model with the distribution produced by the base model through minimization of JensenShannon (JS) divergence [28], thereby establishing a consistent probabilistic space and enhancing robustness against potential miscalibration. FRAP subsequently fuses the aligned predictions through confidence-based weighting to obtain refined predictions that incorporate both strong generalization and domain-specific expertise. These fused predictions serve as reference for assessing predictions from the base model through their consistency with the reference. Extensive experiments across diverse datasets and architectures demonstrate that FRAP generally outperforms representative baselines under distribution shift. In general, our contributions are summarized as follows. • Proposition of FRAP, a novel paradigm for performance estimation on unlabeled datasets under distribution shift. • Design of a pipeline that synergistically integrates the generalization of the foundation model with the domainspecific expertise of the given model. • Deeper investigation and additional experiments are conducted to verify the feasibility of FRAP.
2. Related Works 2.1. Accuracy Estimation Estimating model accuracy on unlabeled out-of-distribution dataset, while conceptually simple, remains highly challenging due to the black-box nature of neural networks and the data-dependent behavior of their predictions. These characteristics make rigorous theoretical modeling difficult, leading existing approaches to rely primarily on empirical heuristics extracted from model outputs. Prior work explores signals based on prediction agreement [1, 19], confidence statistics [6, 11, 43], and prediction-distribution shifts [31, 36, 58, 61]. Despite methodological diversity, these methods share a key limitation: they derive all evidence exclusively from the model under evaluation. This reliance becomes increasingly unreliable under distribution shift due to biased prediction behavior. The approach introduced in this work departs from this paradigm by incorporating a robust foundation model with strong cross-domain generalization to supply auxiliary guidance, allowing more stable estimation signals under shift.
2.2. Source Free Domain Adaptation Unsupervised Domain Adaptation (UDA) aims to improve the generalization of model to a target domain without target labels, typically through distribution alignment [29, 51],
adversarial training [9, 52], or self-training with pseudo labels [57, 63]. Source-Free Domain Adaptation (SFDA) extends this setting by removing access to source data during adaptation [20, 23, 26, 27], making explicit feature alignment infeasible and increasing reliance on pseudo-labeling in the target domain. Recent advances improve pseudo-label quality through generative assumptions [25] or by using generalizable multimodal knowledge such as CLIP [49]. Although conceptually related through the use of external generalization signals, SFDA focuses on adaptation, whereas the objective in this work is performance estimation on unlabeled data. This fundamental difference leads to distinct design principles and methodological requirements.
3. Preliminaries This section introduces the necessary notation and formalizes the performance estimation problem considered in this work. Building on these definitions, we then present the theoretical basis that motivates the FRAP framework.
3.1. Problem Formulation We focus on the standard image classification setting, where each input-label pair corresponds to an image and its associated semantic category. Let X denote the input image space and Y the label space. The source domain follows distribution Ds , while test data are drawn from a different distribution Dt . A classifier fθ trained on Ds , is typically evaluated on Dt , often incurring substantial performance degradation [21, 40]. Here, θ represents the full set of trainable parameters within the neural network classifier fθ . For clarity, this classifier is referred to as the base model. Reliable deployment requires accurate estimation of model performance on unlabeled test data. In this study, we address performance estimation under covariate shift [14, 47], where the label space remains unchanged across domains, i.e., Ys = Yt = {y1 , . . . , yK } where K indicates the number of classes. Given the labeled source dataset, the unlabeled test dataset, and the trained base model, the central question becomes: How can one reliably estimate the performance of this model on the target domain without access to ground-truth labels?
3.2. Reformulation of Accuracy For any input x ∈ X , the base model fθ outputs a predictive distribution Pbθ (· | x) over the K classes, lying in the (K −1)-simplex. The predicted pseudo-label is defined as yb(x) = arg maxj Pbθ (j | x). For target-domain samples {(xi , yi )}N i=1 drawn from Dt , the empirical accuracy is N
ACC(Xt , fθ ) =
1 X I{b y (xi ) = yi }, N i=1
Class
OOD Image
Model 1
0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Base model （domain expertise）
Tiger cat
2
Foundation model (generalization)
0.3
2
3
...
k-1
N
k
0.1 0 1
2
3
...
k-1
k
N
JS divergence
�
0.16 0.14 0.12 0.1 0.08 0.06 0.04 0.02 0
Calibration
0.6 0.5 0.4
. . .
K
0.4
. . .
�r
0.5
0.2
1
Complementary
. . . .
. . . .
. . . . Blackfooted ferret
0.7 0.6
. . .
��
Fused prediction
Confidence Weighted Fusion . . .
Labrador retriever
Prediction
0.3 0.2
N 1
2
3
...
k-1
0.1
N
0
k
Test time Calibration
1
2
3
...
k-1
k
Figure 1. FRAP Overview. FRAP framework leverages a source-trained base model fθ together with a robust foundation model fr . The prediction from the foundation model is first calibrated via temperature scaling τ guided by the output of the base model during test time, and is subsequently fused with the base model prediction through a confidence-weighting scheme. The resulting refined predictive distribution acts as a surrogate for the true label distribution.
where I{·} is the indicator function. By introducing the one-hot ground-truth label distribution P ∗ (· | xi ) ∈ {0, 1}K for sample xi with P ∗ (j | xi ) = I{j = yi }, The expected accuracy can be approximated as following reformulation. N
E[ACC] ≈
K
1 XX b Pθ (j | xi ) P ∗ (j | xi ). N i=1 j=1
(1)
A complete proof is provided in Sec. A. This formulation provides a natural proxy for performance estimation: the predicted distribution is accessible via forward evaluation of the base model, whereas the ground-truth distribution is unknown in the target domain. FRAP addresses this by constructing a fused distribution that serves as an effective surrogate for P ∗ (· | x).
4. Fused Reference Alignment Prediction This section presents the construction of the fused distribution that serves as a surrogate for ground-truth labels. The motivation arises from the observation that the base model encodes valuable domain-specific expertise acquired through task-oriented training [35, 60], yet its predictions become biased or unreliable once the input distribution shifts [16]. Foundation models, on the other hand, demonstrate strong generalization across diverse distributions [2, 8, 50]. These complementary characteristics indicate that integrating the broad generalization of the foundation model with the taskspecific knowledge of the base model can produce a more faithful proxy for the unknown label distribution. FRAP puts this idea into practice by combining predictions from the two models through confidence-based weight-
ing, where calibrated confidence provides a natural indication of prediction reliability. Direct fusion, however, is not effective because the two models output confidence values on incompatible scales due to their differing training paradigms. Furthermore, the foundation models used in this work, i.e., CLIP [38] and SigLIP [62]) typically produce raw cosine similarities that are poorly calibrated for direct probabilistic use. Without proper scaling, these raw scores fail to reflect true predictive accuracy, a challenge that also applies to other potential foundation models. FRAP addresses these issues by applying test-time calibration to align the prediction distribution generated by the foundation model with the distribution produced by the base model before fusion. The aligned output establishes a shared probabilistic space that enables meaningful confidence-weighted integration. An overview of the FRAP workflow is provided in Fig. 1.
4.1. Confidence-Weighted Fusion A simple way to combine two model predictions is to take their uniform average. However, this ignores that confidence scores naturally encode how much each model should be trusted. Under proper calibration, higher confidence should correspond to higher predictive certainty and a greater probability of correctness. Motivated by this, we view fusion as selecting, for each input x, a single distribution P ∗ (· | x) in the PK label probability simplex ∆K = {Q ∈ RK + : j=1 Q(j) = 1} that interpolates between the two model predictions according to their confidence. The confidence associated with the base model and foundation model is defined as cb (x) = max Pbθ (j | x), j
cr (x) = max Pr (j | x), j
Accuracy (ACC)
Confidence (CONF)
Random Baseline
0.4
the probabilities assigned to other labels. This systematic under-confidence severely limits the informativeness as fusion weights and can be attributed to the intrinsic trade-off of contrastive learning objectives [7, 46, 55] between uniformity and alignment, which we provide additional details in Sec. B.
0.2
4.2. Test Time Calibration
0.0
To address the miscalibration issue identified above, we perform test-time calibration on predictions produced by foundation models (e.g. CLIP and SigLIP). A straightforward baseline is to apply a fixed small temperature that rescales the predictive distribution of such foundation models and compensates for suppressed confidence [12]. However, this baseline is inherently data dependent, i.e., an appropriate temperature cannot be chosen without access to labeled target data, and a temperature tuned on labeled source data often fails to transfer under distribution shift. Therefore, we adopt a dynamic calibration scheme that uses the base model as an adaptive reference for foundation models at test time. Concretely, we align the prediction distribution of foundation models with the prediction distribution of the base model by minimizing their divergence, which yields an unsupervised calibration procedure that automatically adapts to the test dataset. Under distribution shift, the base model typically becomes somewhat over-confident [33, 48], yet its prediction distribution still provides a much more informative confidence scale than the nearly uniform probabilities produced by raw similarity of foundation models. The proposed strategy therefore uses the base model as a natural anchor and enforces compatible confidence scales between the two models, which is crucial for effective confidence-weighted fusion. Formally, a temperature parameter τ is introduced to rescale the similarity distribution of foundation models. For an input x, the calibration loss is defined as Lcal (x; τ ) = JS Pbθ (· | x), Pτr (· | x) , (2)
0.8
6 vin g2 no nli
17 ing liv
0 y3 en tit
en tit
y1
10 F 0 Im mow ag e Tin M Net yIm NI ag ST eN et
FA R
CI
FA R CI
3
0.6
10
Accuracy / Confidence
1.0
Dataset
Figure 2. Calibration behavior of raw CLIP similarity scores across diverse datasets. The green dashed line indicates the random baseline corresponding to uniform prediction over the label space (nearly overlapping with the red confidence curve).
and is converted into mixture weights over the two models by wm (x) = P
cm (x) , cm′ (x)
m ∈ {b, r}.
m′ ∈{b,r}
In particular, the weight of the base model is α(x) = wb (x) and the weight of foundation model is 1 − α(x) = wr (x). For notational convenience, let Pb (· | x) = Pbθ (· | x), the fused distribution is then defined implicitly as X 2 P ∗ (· | x) ∈ arg min wm (x) Q(·)−Pm (· | x) 2 , Q(·)∈∆K
m∈{b,r}
and is adopted as a surrogate for the ground-truth label distribution. The effectiveness of this confidence-weighted fusion critically relies on both models being reasonably calibrated. In practice, this assumption is violated for two fundamental reasons. First, the confidence scales of the two models differ substantially due to their distinct training paradigms. The base model, typically trained with supervised crossentropy loss on task-specific data, tends to output sharply peaked probability distributions. In contrast, foundation model is trained with a contrastive objective on large-scale image–text pairs from diverse domains, and consequently produces noticeably flatter predictive distributions. This mismatch in confidence scales can induce biased weighting in the fusion, causing one model to be systematically over- or under-weighted regardless of its actual predictive quality. Second, the raw outputs from foundation models, such as CLIP, are often poorly calibrated for direct use as probabilistic confidence. As illustrated in Fig. 2, when using the raw cosine similarities between image and text embeddings, the resulting predictive distributions are often close to uniform over the label space. Even for correctly classified examples, the top-1 probability is only slightly higher than
where Pτr (· | x) is the temperature-scaled prediction of foundation model and Pbθ (· | x) is the prediction of the base model. The global calibration objective aggregates this loss over the unlabeled test distribution: τ ∗ = arg min Ex∼Dtest Lcal (x; τ ) . τ >0
The temperature-scaled prediction of foundation model that appears in Eq. (2) is given by exp zjr (x)/τ r Pτ (j | x) = PK , r k=1 exp zk (x)/τ where zjr (x) denotes the image-text cosine similarity computed by foundation model between input x and the text
embedding of class j. This test-time calibration procedure learns τ ∗ in a data-driven manner and adaptively mitigates under-confidence in foundation models. The complete pipeline is summarized as pseudo-code in Algorithm 1. Algorithm 1 Fused Reference Alignment Prediction (FRAP) 1: Input: base model fb , foundation model fr , labeled val
dataset Dval , unlabeled test dataset Dtest . 2: Output: estimated error of base model on Dtest . 3: (step 1) Test-time Calibration of fr prediction 4: Obtain pb (x) and zr (x) from fb and fr , respectively. 5: while not converged do P 1 JS pb (xi ) ∥ softmax(zr (xi )/τ ) 6: L ← |B| xi ∈B
7: τ ← τ − η · ∇τ LB 8: end while 9: Calibration result p̃r (x) ← softmax(zr (x)/τ ∗ ). 10: (step 2) Confidence-weighted Fusion 11: Fused prediction p̂(x) ← α pb (x)+(1−α)p̃r (x), where
α is normalized confidence of pb (x) relative to p̃r (x). 12: (step 3) Performance Estimation 13: Compute sample-wise score si = ⟨pb (xi ), p̂(xi )⟩. 14: Determine threshold δ from Dval and estimate error on
Dtest as the portion of samples satisfying si ≤ δ.
5. Experiments In this section, we empirically evaluate FRAP and compare it against competitive baselines. We first train the base model on in-distribution source data with different random seeds and then evaluate the resulting checkpoints on unlabeled out-of-distribution data. The true labels of the test data are only used to compute the estimation error. Datasets. We evaluate our method on a diverse suite of 10 benchmark datasets covering both natural and synthetic distribution shifts, i.e., MNIST [24], CIFAR-10 [22], CIFAR100 [22], ImageNet [44], Tiny-ImageNet, FMoW [5], and four BREEDS datasets [45]. These benchmarks collectively span a wide range of real-world and subpopulation shifts, from digit recognition to large-scale object classification. For each source dataset, we consider its standard in-distribution training set and corresponding out-of-distribution test sets constructed from natural variants (e.g. ImageNet-V2, CIFAR10.1) and corruption benchmarks (e.g. CIFAR-10/100-C, ImageNet-C). Further dataset specifications are provided in Sec. C. Model Architectures. We consider different architectures tailored to each dataset. For MNIST, we construct a lightweight convolutional neural network. For CIFAR10 and CIFAR-100, we utilize DenseNet121 [17] and ResNet18 [13]. For ImageNet, Tiny-ImageNet, FMoW, and BREEDS, we adopt DenseNet121 and ResNet50 [13]. For
each model, we train with three random seeds (i.e. 0, 1, and 10) and save checkpoints at different epochs. Evaluation is conducted across architectures, random seeds, and training epochs to provide a comprehensive assessment of the estimation method. For the foundation model, we employ two vision-language models, CLIP with a ViT-B/32 backbone and SigLIP with a ViT-B/16 backbone, both using publicly available pre-trained weights without fine-tuning.
5.1. Comparison Results We benchmark our method against several representative baselines for accuracy estimation under distribution shifts, including Importance Re-weighting (IM) [3], Average Confidence (AC), Difference of Confidence (DoC) [11], Generalized Disagreement Equality (GDE) [19], Average Thresholded Confidence (ATC) [10], including variants MC and NE, Projection Normalization (ProjNorm) [31, 61], Confidence Optimal Transport (COT) and its thresholded variant COTT [31]. Detailed formulations and implementation specifics for all baselines are provided in Sec. D. We evaluate all methods using Mean Absolute Estimation Error (MAE), defined as the average absolute difference between the true and estimated error rates on the target data. Experiments are conducted across multiple datasets, model architectures at different training epochs, and random seeds for a comprehensive comparison. As summarized in Tab. 1, FRAP based on CLIP achieves the lowest average MAE of 6.53%, outperforming all baseline methods, with the next-best method COTT at 6.72%. While SigLIP yields a slightly higher average error (7.32%), it remains highly competitive against specialized baselines like COT and ATC, demonstrating its robustness across different foundation models. Furthermore, FRAP achieves the best result on 6 out of the 18 individual datasets, demonstrating superior and robust estimation performance. Regarding computational cost, COT/COTT scales quadratically or cubically with the number of classes due to the optimal transport computation. FRAP, in contrast, requires foundation model inference that scales linearly with the number of classes and avoids the expensive OT solver, making it substantially more efficient in high-cardinality label spaces.
5.2. Calibration with Different Temperature FRAP calibrates the predictions of foundation models using the dynamically learned temperature introduced in Sec. 4.2. As shown in Tab. 2, our test-time calibration (TTC) effectively alleviates the miscalibration of the original similarity (Raw) in most settings. By reducing the Expected Calibration Error (ECE), TTC substantially narrows the gap between confidence scores and actual performance. We further explore the effect of fixed temperatures by adopting three constant values (i.e. 0.1, 0.05, 0.01) and compare the calibration errors between our test-time-learned tem-
Table 1. Comparison of FRAP and baselines. Mean Absolute Error (MAE)(%) ↓ across different methods and datasets. FRAP is evaluated using both CLIP and SigLIP as foundation models. Results reported by aggregating MAE numbers over different seeds and architectures, shown as mean (%) ± standard deviation.
Baselines Ours (FRAP) Dataset IM AC DoC GDE ATC-MC ATC-NE projNorm COT COTT SigLIP CLIP MNIST 6.86±.12 1.97±.02 1.96±.02 25.05±.27 10.27±.10 13.80±.11 14.77±.18 6.04±.06 13.81±.09 10.56±.10 12.42±.10 C10-N 11.55±.05 11.54±.05 11.54±.05 7.04±.04 12.00±.07 12.67±.06 9.97±.06 11.17±.05 11.96±.07 7.54±.03 2.14±.02 C10-S 14.24±.08 14.17±.08 14.16±.08 8.62±.07 14.11±.09 14.23±.09 5.14±.03 8.99±.03 9.56±.06 7.68±.06 3.24±.05 C100-N 13.06±.08 13.02±.08 13.15±.08 11.72±.07 12.65±.06 13.24±.06 10.73±.10 12.37±.08 12.72±.06 10.22±.04 4.00±.03 C100-S 21.30±.11 21.26±.08 21.39±.11 16.94±.04 19.48±.10 19.62±.11 12.33±.07 13.73±.08 13.66±.06 17.87±.10 11.42±.10 IN-S 11.09±.07 8.14±.06 8.96±.06 7.73±.05 2.57±.02 4.07±.03 4.98±.03 2.67±.02 2.14±.02 3.74±.02 4.76±.01 IN-N 7.10±.08 5.81±.06 6.28±.07 6.27±.05 2.45±.02 1.48±.01 8.76±.05 3.38±.02 1.91±.01 1.15±.01 2.16±.03 IN200-S 11.53±.09 7.99±.08 9.06±.08 30.22±.13 3.36±.05 6.86±.04 4.96±.04 3.02±.02 4.38±.02 6.14±.05 9.98±.04 IN200-N 14.56±.16 12.51±.14 12.87±.15 31.28±.24 7.17±.09 4.17±.05 11.33±.10 7.04±.07 3.50±.05 5.43±.07 7.46±.04 FMoW 3.66±.04 4.00±.04 3.63±.04 6.04±.04 2.60±.03 2.33±.03 22.24±.10 8.02±.05 8.76±.06 2.70±.03 3.42±.02 L17-S 9.97±.08 8.96±.08 9.16±.08 9.11±.10 5.35±.06 6.86±.06 6.81±.05 3.44±.03 4.09±.03 6.81±.06 8.55±.05 L17-N 23.55±.05 22.11±.06 22.43±.06 18.38±.10 11.11±.07 9.26±.06 16.91±.08 13.94±.04 7.75±.05 6.74±.07 5.37±.05 NL26-S 11.63±.11 9.54±.10 10.01±.10 12.63±.14 6.87±.09 7.98±.08 6.54±.05 3.18±.02 3.72±.03 7.60±.09 9.30±.07 NL26-N 22.55±.08 19.46±.08 20.27±.08 21.65±.12 9.70±.10 8.85±.09 12.01±.08 11.34±.03 5.83±.04 8.48±.10 6.66±.09 E13-S 12.27±.10 10.95±.09 11.33±.10 11.54±.10 5.91±.07 6.18±.06 6.10±.05 4.14±.03 3.14±.03 5.78±.07 8.29±.05 E13-N 20.97±.08 19.25±.08 19.79±.08 17.98±.10 10.70±.07 8.39±.07 12.75±.06 10.22±.04 5.76±.03 8.50±.07 5.90±.06 E30-S 13.61±.11 10.93±.10 11.67±.10 9.92±.11 5.62±.08 5.28±.06 7.50±.06 3.85±.03 2.30±.02 5.81±.08 6.94±.06 E30-N 23.30±.08 19.87±.08 20.89±.08 15.50±.10 10.10±.08 7.33±.07 14.52±.09 11.15±.03 5.93±.03 9.05±.09 5.54±.07 Average 13.76 12.30 12.70 14.87 8.45 8.48 10.46 7.65 6.72 7.32 6.53
VLM CLIP CLIP
Method C10 C100 FMoW ImgN MNIST TinyImg E13 TTC Raw
SigLIP TTC SigLIP Raw
0.128 0.150 0.590 0.373 0.159 0.209 0.632 0.478
0.018 0.116 0.034 0.122
0.127 0.343 0.171 0.427
0.073 0.027 0.283 0.510
0.061 0.384 0.092 0.456
E30
L17 NL26
0.295 0.321 0.540 0.427 0.517 0.474 0.632 0.557 0.281 0.348 0.419 0.422 0.501 0.499 0.668 0.567
perature and these fixed temperature coefficients. As shown in Fig. 3, regarding the calibration error (ECE), our test-timelearned temperature (TTC) predominantly achieves superior results compared to fixed temperature scaling at τ = 0.05 and τ = 0.1, although the fixed small temperature τ = 0.01 yields the lowest calibration error in several cases. When the calibrated CLIP distributions are used for performance estimation, our standard FRAP with dynamically learned temperature significantly outperforms FRAPτ =0.05 (6.53% vs. 8.19%), and marginally outperforms FRAPτ =0.01 (6.53% vs. 6.61%). The full results are reported in Tab. 3, where RAP denotes the method without temperature-scaling calibration, FRAPτ =0.05 and FRAPτ =0.01 denote calibration with fixed temperatures 0.05 and 0.01, respectively, and FRAPdyna denotes our standard calibration with test-timelearned temperature. Although fixed small temperature τ = 0.01 achieve lower calibration error, it leads to slightly worse estimation performance than TTC. The results indicate that improved calibration, as mea-
Comparison: fixed-temperature vs dynamic-temperature(TTC) 0.8
Expected Calibration Error (ECE)
Table 2. Expected Calibration Error (ECE) ↓ comparison before and after TTC. The results across various datasets demonstrate that TTC significantly reduces ECE for both CLIP and SigLIP reference models.
T=0.1 T=0.05 T=0.01 dynamic
0.7 0.6
0.589 0.548
0.5
0.514
0.488
0.4
0.3670.356
0.540 0.455 0.423
0.427
0.376 0.356
0.3420.338
0.295
0.3
0.350
0.321
0.191
0.2
0.150
0.128
0.1 0.0
0.533 0.492
0.487 0.444
10
R CIFA
0.127 0.073
0.056
0.047
0.0140.003
100
R CIFA
0.105 0.083
0.0470.061 0.006
0.007
0.037
0.012
0.024
T 7 6 3 0 et et MNIS ImageN ImageN living1 entity1 entity3 nliving2 no Tiny
0.018
w
Fmo
Figure 3. Fixed temperature vs. Test-time calibration. We compare the ECE of our test-time-learned temperature (TTC) against the fixed temperature values (τ = 0.1, 0.05, 0.01) across diverse datasets, with CLIP serving as the foundation model.
sured by ECE, does not necessarily translate into improved estimation performance. We attribute this discrepancy to a fundamental mismatch between the calibration objective and the downstream estimation objective. Small fixed temperatures attain lower miscalibration error, while they are not optimized for our goal of enabling effective confidence-weighted fusion with the base model. The dynamically learned temperature, chosen to align the predictions of foundation model with those of the base model, implicitly acts as a regularizer that enforces compatible confidence scales between the two models and preserves the utility of the fusion scheme. This perspective explains why our approach, despite exhibiting higher miscalibration error than the τ = 0.01 variants, ultimately yields superior performance estimation.
Table 3. Comparison of MAE under different temperature. Results are reported by aggregating MAE numbers over different seeds and architectures, shown as mean (%) ± standard deviation. Fixed temperature TTC Dataset RAP FRAPτ =0.05 FRAPτ =0.01 FRAPdyna MNIST 26.31±0.12 11.30±0.10 11.13±0.10 12.42±0.10 C10-N 7.26±0.02 10.86±0.04 8.48±0.03 2.14±0.02 C10-S 7.47±0.07 12.95±0.07 8.01±0.05 3.24±0.05 C100-N 12.24±0.06 12.57±0.06 10.90±0.05 4.00±0.03 C100-S 7.15±0.06 19.56±0.10 17.07±0.10 11.42±0.10 IN-S 6.93±0.03 2.94±0.02 3.11±0.02 4.76±0.01 IN-N 1.61±0.01 1.99±0.02 1.43±0.01 2.16±0.03 IN200-S 16.20±0.07 4.34±0.04 4.95±0.04 9.98±0.04 IN200-N 2.85±0.02 6.12±0.07 5.32±0.06 7.46±0.04 FMoW 8.10±0.05 2.51±0.03 3.39±0.03 3.42±0.02 L17-S 11.71±0.07 5.74±0.06 6.15±0.05 8.55±0.05 L17-N 6.92±0.04 10.57±0.07 5.70±0.04 5.37±0.05 NL26-S 6.76±0.03 7.20±0.09 5.64±0.05 9.30±0.07 NL26-N 4.61±0.03 9.14±0.09 5.74±0.05 6.66±0.09 E13-S 5.11±0.04 5.73±0.06 4.93±0.05 8.29±0.05 E13-N 6.92±0.04 9.43±0.07 6.98±0.06 5.90±0.06 E30-S 4.69±0.03 5.42±0.07 4.07±0.05 6.94±0.06 E30-N 4.74±0.03 9.05±0.08 6.01±0.05 5.54±0.07 Average 8.20 8.19 6.61 6.53
5.3. Effectiveness of Confidence-Weighted Fusion To further assess the effectiveness of the confidenceweighted fusion strategy, we introduce the Semantic Alignment Score (SAS), which measures the semantic consistency between predictions and ground-truth labels. This analysis is conducted on several datasets constructed based on WordNet [32], a human-curated lexical database that organizes concepts into a tree-structured taxonomy. Following [37, 56], the semantic proximity of two concepts can be measured by taxonomic distance d(·, ·) denoting the shortest-path edge count between them, path similarity spath (·, ·) as the inverse of d(·, ·), and Wu-Palmer similarity swup (·, ·) based on the depth of their lowest common ancestor relative to the depths of the individual concepts. We then compute the comprehensive semantic similarity between ci and cj as S(ci , cj ) = swup (ci , cj ) + spath (ci , cj ) − d(ci , cj ).
(3)
Given a predicted probability distribution p = [p1 , . . . , pK ] ∈ RK over K classes and the ground-truth label y ∈ {1, . . . , K}, SAS is defined as SAS(p, y) =
K X
pi · S(ci , cy ),
As shown in Fig. 4, we compute the SAS gap ∆SAS = SASfusion − SASbaseline on ImageNet and its variants, where the baseline is either the base model or CLIP. Under natural distribution shifts, illustrated in Fig. 4a, confidence-weighted fusion consistently achieves positive ∆SAS values across nearly all experiments. Under synthetic corruptions, illustrated in Fig. 4b, fusion maintains predominantly positive differences across the majority of test cases, indicating that the proposed strategy produces predictions that are more semantically coherent with ground-truth labels. We conduct the same analysis on additional datasets including Tiny-ImageNet, Living-17, and Nonliving-26. The results show that confidence-weighted fusion generally improves semantic alignment, with particularly notable gains on Living-17 and Nonliving-26. Detailed results and analysis are provided in Sec. E.
5.4. Thresholding: Bridge the Gap to the Ideal The fused predictions provide a practical surrogate in the absence of ground truth, and this approximation is inherently imperfect and inevitably deviates from the ideal one-hot labels, which leads to non-negligible estimation errors. To compensate for this gap, we introduce a simple yet effective thresholding strategy inspired by [10]. Concretely, let Est(x) denote the estimation score that serves as an approximation of the accuracy, computed as in Eq. (1), where the ideal label distribution is approximated by the fusion of the two predictive models. We define a threshold δ on the source validation set Ds such that the fraction of samples with Est(x) below t matches the error rate of fθ : X 1 1 X I{b y (x) ̸= y}, I{Est(x) < δ} = |Ds | |Ds | x∈Ds
(x,y)∈Ds
where yb(x) = arg maxj Pbθ (j | x) and Pbθ (· | x) is the predictive distribution of the base model fθ . This strategy effectively reduces the problem from estimating absolute score values to making binary correctness decisions. Rather than attempting to perfectly approximate the ideal label distribution, the estimator only needs to decide whether the score lies above or below the threshold. Since Est(·) is derived from the fusion of the two predictive models, it inherently combines their complementary strengths. By applying the same threshold δ to the unlabeled test set, the proportion of samples with Est(x) < δ yields the estimated error rate of the classifier under distribution shift in our method.
i=1
5.5. Reference Model Dependency and Robustness
where pi is the predicted probability for class ci , and S(ci , cy ) measures the semantic similarity between class ci and the ground-truth class cy as in Eq. (3). Higher SAS indicates better semantic alignment with the true label, even when the top-1 prediction is incorrect.
To further investigate the source of performance gains and the robustness of our framework, we conduct an extensive analysis across different reference sources. Methodological Gain vs. External Model Benefits. A natural concern is whether the performance improvement of
Fusion vs. CLIP
Improvement: 100.0% Mean: 0.249
Fusion vs. Base Model
Improvement: 98.6% Mean: 0.486
Similarity Gap
0.6
0.6 0.4
0.4
0.2 0.0
0.2
0.2
Fusion vs. CLIP
Improvement: 64.4% Mean: 0.102
1.5
Similarity Gap
Fusion vs. Base Model
Improvement: 98.1% Mean: 0.797
1.0
1.0
0.5
0.5
0.0 0.5
0.0
0.4
0.0
0.6
0
10
20
30
40
50
Test Sample Index
60
70
0
10
20
30
40
50
Test Sample Index
60
70
1.5
1.0
0.5 1.5
0
250 500 750 1000 1250 1500 1750
Test Sample Index
(a) Natural shift
0
250 500 750 1000 1250 1500 1750
Test Sample Index
(b) Synthetic shift
Figure 4. Effectiveness of CWF. Per-experiment Semantic Alignment Score (SAS) gaps on ImageNet. We compare the confidence-weighted fusion against the base model (left column) and CLIP (right column) under (a) natural shifts and (b) synthetic corruptions. Positive similarity gap ∆SAS = SASfusion − SASbaseline indicate improved semantic alignment (∆SAS > 0) and the fusion predominantly achieves positive differences, validating its effectiveness in preserving semantic coherence. Table 4. Extensive baselines. MAE↓ on CLIP , SigLIP , and Random reference for Base (pseudo-labels from the reference), Fix (FRAP with fixed scaling), TTC (FRAP with Test-timecalibration), and Random (FRAP with a random reference). Base
Fix
TTC
Base
Fix
TTC Random
MAE(avg) 10.32
6.61
6.53
9.21
5.65
7.32
8.10
FRAP stems solely from the inclusion of powerful foundation models. We introduce a baseline (Base) that directly employs the pseudo-labels from the reference model as the ground-truth for accuracy estimation. As shown in Tab. 4, FRAP with test-time calibration (Dyna) and its fixedtemperature calibration variant (Fix) consistently outperform the Base method under various benchmark settings (CLIP and SigLIP). This gap demonstrates that FRAP benefits from our proposed framework rather than merely relying on the zero-shot capability of reference model. Robustness under Reference Failure. To test the robustness of FRAP for reference model, we conduct a stress test using a Random reference generated via a Dirichlet distribution. As reported in Tab. 4, FRAP degrades gracefully under this pathological setting instead of failing catastrophically. This demonstrates the architectural-agnostic robustness even when the reference quality is severely compromised. TTC for Reference-Agnostic Reliability. As demonstrated in Tab. 4, foundation models like CLIP and SigLIP exhibit impressive performance when their internal scaling parameters (e.g., τ ≈ 0.01 for CLIP ViT-B/32) are applied (Fix). In particular, Fix (SigLIP) reaches a new SOTA in estimation accuracy across several benchmarks. However, we argue that our proposed Test-Time Calibration (TTC) remains essential for the FRAP framework. One fundamental value of TTC lies in removal of fragile reliance on perfectly calibrated reference models, enabling reliable estimation even when the reference is miscalibrated, or entirely broken. Our Random stress tests confirm that TTC is indispensable when employing smaller, non-foundation networks as references
to save computational costs. Selection of Reference Model. While our framework is reference-agnostic, the choice of reference inevitably influences the estimation ceiling. How to systematically select the most suitable reference model remains an open question, which we leave for future research.
6. Conclusion This work introduces Fused Reference Alignment Prediction (FRAP), a framework for estimating model performance on unlabeled test data under distribution shift by jointly exploiting a task-specialized base model and a broadly trained foundation model. FRAP first reformulates accuracy as an average inner product between the base model predictions and the underlying label distributions, and then approximates the latter with a fused reference distribution. This reference is obtained by test-time calibration of the foundation model via Jensen–Shannon divergence minimization, followed by confidence-weighted fusion that integrates cross-domain generalization from the foundation model with domain-specific expertise from the base model. Combined with a simple thresholding scheme, FRAP achieves average lower mean absolute estimation error than representative baselines across diverse benchmark settings, and remains computationally efficient in high-cardinality label spaces. There are several limitations and directions for further investigation of our work. FRAP does not consistently outperform baselines across all benchmarks as the effectiveness correlates with the generalization limits of the foundation model. These observations point to promising avenues for future work, including developing domain-adapted or taskspecific reference models, extending FRAP to other modalities such as natural language via strong language foundations, and designing fusion and calibration objectives that are more tightly coupled to the performance-estimation goal.
Acknowledgments This work was supported partially by the NSFC General Program (62576367), the NSFC Excellent Young Scientists Fund (Overseas) (2025HY00260105, 2025HYSPT0708), and the Scientific Research Startup Fund at Sun Yat-sen University (25hytd012).
References [1] Christina Baek, Yiding Jiang, Aditi Raghunathan, and J. Zico Kolter. Agreement-on-the-line: Predicting the performance of neural networks under distribution shift. In Advances in Neural Information Processing Systems (NeurIPS), pages 19274–19289, 2022. 1, 2 [2] Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ B. Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen Creel, Jared Quincy Davis, Dorottya Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren E. Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, and et al. On the opportunities and risks of foundation models. CoRR, abs/2108.07258: 1–161, 2021. 3 [3] Mayee F. Chen, Karan Goel, Nimit Sharad Sohoni, Fait Poms, Kayvon Fatahalian, and Christopher Ré. Mandoline: Model evaluation under distribution shift. In International Conference on Machine Learning (ICML), pages 1617–1629, 2021. 5, 3 [4] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML), pages 1597–1607, 2020. 1 [5] Gordon A. Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6172–6180, 2018. 5, 2 [6] Weijian Deng, Yumin Suh, Stephen Gould, and Liang Zheng. Confidence and dispersity speak: Characterising prediction matrix for unsupervised accuracy estimation. In International Conference on Machine Learning (ICML), pages 7658–7674, 2023. 1, 2 [7] Georgiana Dinu, Corey D. Barrett, Yi Xiang, Miguel Romero Calvo, Anna Currey, and Xing Niu. Effective post-training embedding compression via temperature control in contrastive training. In International Conference on Learning Representations (ICLR), pages 1–10, 2025. 4, 2 [8] Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt. Data
determines distributional robustness in contrastive language image pre-training (CLIP). In International Conference on Machine Learning (ICML), pages 6216–6234, 2022. 3 [9] Yaroslav Ganin and Victor S. Lempitsky. Unsupervised domain adaptation by backpropagation. In International Conference on Machine Learning (ICML), pages 1180–1189, 2015. 2 [10] Saurabh Garg, Sivaraman Balakrishnan, Zachary Chase Lipton, Behnam Neyshabur, and Hanie Sedghi. Leveraging unlabeled data to predict out-of-distribution performance. In International Conference on Learning Representations (ICLR), pages 1–10, 2022. 1, 5, 7, 4 [11] Devin Guillory, Vaishaal Shankar, Sayna Ebrahimi, Trevor Darrell, and Ludwig Schmidt. Predicting with confidence on unseen distributions. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 1114–1124, 2021. 1, 2, 5, 3 [12] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning (ICML), pages 1321–1330, 2017. 1, 4 [13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016. 5 [14] James J Heckman. Sample selection bias as a specification error. Econometrica, 47(1):153–161, 1979. 2 [15] Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Representations (ICLR), pages 1–9, 2017. 3 [16] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 8320–8329, 2021. 3 [17] Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2261–2269, 2017. 5 [18] Jonathan J. Hull. A database for handwritten text recognition research. IEEE Trans. Pattern Anal. Mach. Intell., 16(5): 550–554, 1994. 3 [19] Yiding Jiang, Vaishnavh Nagarajan, Christina Baek, and J. Zico Kolter. Assessing generalization of SGD via disagreement. In International Conference on Learning Representations (ICLR), pages 1–9, 2022. 1, 2, 5, 3 [20] Youngeun Kim, Donghyeon Cho, Kyeongtak Han, Priyadarshini Panda, and Sungeun Hong. Domain adaptation without source data. IEEE Trans. Artif. Intell., 2(6):508–518, 2020. 2 [21] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei
Guo, Berton Earnshaw, Imran S. Haque, Sara M. Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning (ICML), pages 5637–5664, 2021. 1, 2 [22] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, Toronto, Ontario, 2009. 5, 2 [23] Jogendra Nath Kundu, Naveen Venkat, Rahul M. V., and R. Venkatesh Babu. Universal source-free domain adaptation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4543–4552, 2020. 2 [24] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proc. IEEE, 86(11):2278–2324, 1998. 5, 2 [25] Jonghyun Lee, Dahuin Jung, Junho Yim, and Sungroh Yoon. Confidence score for source-free unsupervised domain adaptation. In International Conference on Machine Learning (ICML), pages 12365–12377, 2022. 2 [26] Rui Li, Qianfen Jiao, Wenming Cao, Hau-San Wong, and Si Wu. Model adaptation: Unsupervised domain adaptation without source data. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9638–9647, 2020. 2 [27] Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning (ICML), pages 6028–6039, 2020. 2 [28] Jianhua Lin. Divergence measures based on the shannon entropy. IEEE Trans. Inf. Theory, 37(1):145–151, 1991. 2 [29] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. Learning transferable features with deep adaptation networks. In International Conference on Machine Learning (ICML), pages 97–105, 2015. 2 [30] Shangyun Lu, Bradley Nott, Aaron Olson, Alberto Todeschini, Hossein Vahabi, Yair Carmon, and Ludwig Schmidt. Harder or different? a closer look at distribution shift in dataset reproduction. In ICML Workshop on Uncertainty and Robustness in Deep Learning, pages 1–15, 2020. 3 [31] Yuzhe Lu, Yilong Qin, Runtian Zhai, Andrew Shen, Ketong Chen, Zhenlin Wang, Soheil Kolouri, Simon Stepputtis, Joseph Campbell, and Katia P. Sycara. Characterizing out-of-distribution error via optimal transport. In Advances in Neural Information Processing Systems (NeurIPS), pages 17602–17622, 2023. 1, 2, 5, 4 [32] George A. Miller. Wordnet: A lexical database for english. Commun. ACM, 38(11):39–41, 1995. 7 [33] Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks. In Advances in Neural Information Processing Systems (NeurIPS), pages 15682–15694, 2021. 4 [34] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, pages 1–8, 2011. 3
[35] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Trans. Knowl. Data Eng., 22(10):1345–1359, 2010. 3 [36] Ori Press, Ravid Shwartz-Ziv, Yann LeCun, and Matthias Bethge. The entropy enigma: Success and failure of entropy minimization. In International Conference on Machine Learning (ICML), pages 41064–41085, 2024. 1, 2 [37] Roy Rada, Hafedh Mili, Ellen Bicknell, and Maria Blettner. Development and application of a metric on semantic nets. IEEE Trans. Syst. Man Cybern., 19(1):17–30, 1989. 7 [38] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), pages 8748–8763, 2021. 3, 1 [39] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do CIFAR-10 classifiers generalize to CIFAR-10? CoRR, abs/1806.00451:1–13, 2018. 3 [40] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning (ICML), pages 5389–5400, 2019. 1, 2 [41] Joshua Robinson, Li Sun, Ke Yu, Kayhan Batmanghelich, Stefanie Jegelka, and Suvrit Sra. Can contrastive learning avoid shortcut solutions? In Advances in Neural Information Processing Systems (NeurIPS), pages 4974–4986, 2021. 1 [42] Mélanie Roschewitz and Ben Glocker. Distance matters for improving performance estimation under covariate shift. In IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pages 4551–4561, 2023. 1 [43] Elan Rosenfeld and Saurabh Garg. (almost) provable error bounds under distribution shift via disagreement discrepancy. In Advances in Neural Information Processing Systems (NeurIPS), pages 28761–28784, 2023. 1, 2 [44] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. Int. J. Comput. Vis., 115(3):211–252, 2015. 5, 2 [45] Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. BREEDS: benchmarks for subpopulation shift. In International Conference on Learning Representations (ICLR), pages 1–9, 2021. 5, 2 [46] Simon Schrodi, David T. Hoffmann, Max Argus, Volker Fischer, and Thomas Brox. Two effects, one trigger: On the modality gap, object bias, and information imbalance in contrastive vision-language models. In International Conference on Learning Representations (ICLR), pages 1–10, 2025. 4, 2 [47] Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log-likelihood function. J. Stat. Plan. Inference, 90(2):227–244, 2000. 2 [48] Jasper Snoek, Yaniv Ovadia, Emily Fertig, Balaji Lakshminarayanan, Sebastian Nowozin, D. Sculley, Joshua V. Dillon, Jie Ren, and Zachary Nado. Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset
shift. In Advances in Neural Information Processing Systems (NeurIPS), pages 13969–13980, 2019. 1, 4 [49] Song Tang, Wenxin Su, Yan Gan, Mao Ye, Jianwei Zhang, and Xiatian Zhu. Proxy denoising for source-free domain adaptation. In International Conference on Learning Representations (ICLR), pages 1–10, 2025. 2 [50] Weijie Tu, Weijian Deng, and Tom Gedeon. A closer look at the robustness of contrastive language-image pre-training (CLIP). In Advances in Neural Information Processing Systems (NeurIPS), pages 13678–13691, 2023. 3 [51] Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. CoRR, abs/1412.3474:1–8, 2014. 2 [52] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2962–2971, 2017. 2 [53] Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR, abs/1807.03748:1–9, 2018. 1 [54] Haohan Wang, Songwei Ge, Zachary C. Lipton, and Eric P. Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems (NeurIPS), pages 10506–10518, 2019. 3 [55] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning (ICML), pages 9929–9939, 2020. 4, 1, 2 [56] Zhibiao Wu and Martha Stone Palmer. Verb semantics and lexical selection. In Association for Computational Linguistics (ACL), pages 133–138, 1994. 7 [57] Qizhe Xie, Minh-Thang Luong, Eduard H. Hovy, and Quoc V. Le. Self-training with noisy student improves imagenet classification. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684–10695, 2020. 2 [58] Renchunzi Xie, Hongxin Wei, Lei Feng, Yuzhou Cao, and Bo An. On the importance of feature separability in predicting out-of-distribution error. In Advances in Neural Information Processing Systems (NeurIPS), pages 27783–27800, 2023. 1, 2 [59] Chhavi Yadav and Léon Bottou. Cold case: The lost MNIST digits. In Advances in Neural Information Processing Systems (NeurIPS), pages 13443–13452, 2019. 3 [60] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Advances in Neural Information Processing Systems (NeurIPS), pages 3320–3328, 2014. 3 [61] Yaodong Yu, Zitong Yang, Alexander Wei, Yi Ma, and Jacob Steinhardt. Predicting out-of-distribution error with the projection norm. In International Conference on Machine Learning (ICML), pages 25721–25746, 2022. 1, 2, 5, 4 [62] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 11941– 11952, 2023. 3
[63] Yang Zou, Zhiding Yu, B. V. K. Vijaya Kumar, and Jinsong Wang. Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In European Conference on Computer Vision (ECCV), pages 297–313, 2018. 2
Bridging Domain Expertise and Generalization for Performance Estimation Supplementary Material A. Derivation of Proposition 1 For a finite target sample set {(xi , yi )}N i=1 , the standard empirical accuracy (i.e., Hard Accuracy) of the model fθ is defined as:
Finally, approximating the outer integral with the finite target samples {xi }N i=1 via Monte Carlo estimation yields: N
E[ACCsoft ] ≈
N
ACCN =
1 X I{b y (xi ) = yi }, N i=1
(A.1)
where yb(x) = arg maxj Pbθ (j | x) is the hard label prediction. Since the hard indicator function is non-differentiable and difficult to analyze directly, we consider the expected soft accuracy as a surrogate for theoretical analysis. The soft accuracy replaces the hard match with the probability of the true class: N
ACCsoft N =
1 Xb Pθ (yi | xi ). N i=1
(A.2)
N →∞
(A.3) Using the law of iterated expectations (conditioning on X), we expand this term as: Z E[ACCsoft ] = EY |x Pbθ (Y | x) pt (x) dx. (A.4) X
Inside the integral, the term EY |x [Pbθ (Y | x)] represents the expected confidence score of the model with respect to the label variable Y given X = x. By the definition of expectation for a discrete random variable, this expectation expands to the sum of the function values weighted by their probabilities: K X EY |x Pbθ (Y | x) = P ∗ (j | x) · Pbθ (j | x) , (A.5) | {z } | {z } True Probability Model Prediction
where P ∗ (j | x) ≜ P(Y = j | X = x) denotes the groundtruth class posterior, and Pbθ (j | x) denotes the predictive probability of the model for class j. Substituting this expansion back into Eq. (A.4), we obtain: Z K X E[ACCsoft ] = Pbθ (j | x) P ∗ (j | x) pt (x) dx. X
B. Contrastive representation learning Many contrastive embedding models are trained using the InfoNCE (Information Noise-Contrastive Estimation) objective [4, 38, 41, 53, 55]. Fundamentally, this loss shapes the embedding space by pulling representations of positive (matched) pairs closer while pushing negative (unmatched) pairs apart. Formally, given a mini-batch of N samples, for each query sample qi , the InfoNCE loss is defined as: N
By the Law of Large Numbers (LLN), as N → ∞, the empirical soft accuracy converges to its population expectation over the target distribution Dt : b lim ACCsoft N = E(X,Y )∼Dt Pθ (Y | X) ≜ E[ACCsoft ].
j=1
K
1 XX b Pθ (j | xi ) P ∗ (j | xi ). (A.7) N i=1 j=1
j=1
(A.6)
exp(sim(qi , k+ 1 X i )/τ ) log PN , N i=1 j=1 exp(sim(qi , kj )/τ ) (B.1) d where qi , k+ ∈ R denote the embeddings of the query i and its corresponding positive key, and {kj }N j=1 includes + the positive key ki and N − 1 negative keys. The function sim(u, v) = u⊤ v/(∥u∥∥v∥) computes cosine similarity, and τ ∈ R+ is a temperature parameter controlling the concentration of the distribution. LInfoNCE = −
B.1. CLIP Training CLIP applies InfoNCE to learn aligned image-text representations. Given a batch of N image-text pairs {(Ii , Ti )}N i=1 , CLIP encodes images and texts into ℓ2 -normalized embeddings zvi , zti ∈ Rd using vision encoder fv and text encoder ft respectively: zvi =
fv (Ii ) , ∥fv (Ii )∥2
zti =
ft (Ti ) . ∥ft (Ti )∥2
(B.2)
The symmetric contrastive loss is calculated as the average of the image-to-text (LI→T ) and text-to-image (LT →I ) losses. For the i-th sample, the image-to-text loss is: exp(sim(zvi , zti )/τ ) LI→T = − log PN , i v t j=1 exp(sim(zi , zj )/τ )
(B.3)
where sim(zvi , ztj ) = (zvi )⊤ ztj is the cosine similarity. Symmetrically, the text-to-image loss is: exp(sim(zti , zvi )/τ ) LTi →I = − log PN . t v j=1 exp(sim(zi , zj )/τ )
(B.4)
The final CLIP loss is the average of both directions: N
LCLIP =
1 X I→T Li + LTi →I , 2N i=1
(B.5)
B.2. Mechanism of Uniformity To understand why CLIP yields predictions with high uniformity (i.e., over-smoothed distributions), we analyze the InfoNCE loss through two complementary lenses: AlignmentUniformity decomposition and Gradient dynamics. Alignment and Uniformity. Previous work [55] decomposes the quality of learned representations into two key properties: • Alignment measures the closeness of positive pairs. • Uniformity quantifies how uniformly features are distributed on the hyper-sphere. InfoNCE implicitly optimizes both objectives, but the balance between them evolves during training. In the early stage (i.e. Alignment phase), the model rapidly pulls positive pairs closer and the numerator term in InfoNCE dominates optimization, as positive similarities are initially low. In the late stage (i.e. Uniformity phase), once positive pairs are sufficiently aligned, the numerator approaches its maximum and the gradient signal from the numerator nearly diminishes. Thus the optimization shifts to minimizing the denominator of the InfoNCE loss, which encourages spreading out negative pairs uniformly across the hypersphere [46]. Consequently, the resulting prediction probabilities tend to spread out over all classes rather than concentrating on a single peak, especially when semantic ambiguity exists. Implicit Hard Negative Mining. Another explanation takes a different but consistent perspective that the InfoNCE performs implicit hard negative mining via its exponential weighting scheme [7]: ∂LInfoNCE ∝ exp(sim(qi , kj )/τ ), ∂sim(qi , kj )
j ̸= i,
(B.6)
which means that hard negatives receive exponentially larger gradients, causing the model to focus more on distinguishing confusing negative samples. Consequently, InfoNCE automatically emphasizes learning from the most challenging examples without explicit hard negative mining. Let us derive the gradient of InfoNCE loss, which is an equivalent form of the formulation Eq. (B.1): Li = − log
exp(s+ /τ ) , Pi + exp(si /τ ) + j̸=i exp(s− j /τ )
(B.7)
− where s+ i denotes the positive pair similarity and sj denotes the negative pair similarity. For a negative pair similarity s− j ,
the gradient is: " # exp(s+ ∂ ∂Li i /τ ) = − − log P − ∂s− ∂sj exp(s+ j i /τ ) + k̸=i exp(sk /τ ) + X s ∂ i . = − log exp(s+ exp(s− i /τ ) + k /τ ) − τ ∂sj k̸=i
=
exp(s− 1 j /τ ) . P + τ exp(si /τ ) + k̸=i exp(s− k /τ )
(B.8)
Therefore, the gradient magnitude is proportional to the exponential term: ∂Li ∝ exp(s− j /τ ), ∂s− j
(B.9)
which shows that negative samples with higher similarity (hard negatives) receive larger gradients. This implicit weighting mechanism naturally emphasizes hard negatives during optimization, leading to more uniform feature distribution to separate these challenging cases on the hypersphere, as discussed in [7, 46]. This training dynamic is fundamentally intertwined with the temperature parameter τ . Since CLIP trains a learnable temperature which typically converges to a very low value (τ ≈ 0.01), this low τ imposes severe uniformity pressure on the feature space by aggressively magnifying the gradient signal from hard negatives. Though low τ mathematically causes the softmax output to be sharper (high confidence) by magnifying logit differences, the resultant high feature uniformity is the root cause of the observed calibration issue: it leads to overly uniform prediction distributions for inputs lacking strong semantic alignment, as we observe in CLIP.
C. Datasets We focus on natural image classification and evaluate our method on 10 benchmark datasets encompassing both natural and synthetic distribution shifts. Specifically, we use MNIST [24], CIFAR-10 [22], CIFAR-100 [22], ImageNet [44], Tiny-ImageNet, FMoW [5], and four datasets from the BREEDS benchmark [45]: Living-17, Nonliving26, Entity-13, and Entity-30. Tiny-ImageNet is a compact subset of ImageNet comprising 200 classes. BREEDS (Benchmark for Robustness under Evolving Distribution Shifts) constructs subpopulation shifts by partitioning ImageNet classes hierarchically, where training and test sets contain different fine-grained subclasses within the same superclass, (e.g. different dog breeds), simulating realistic deployment scenarios. For each source dataset, we evaluate performance on corresponding shifted test sets under distribution shift.
Table 5. Summary of datasets and their corresponding distribution shifts. For CIFAR, ImageNet, Tiny-ImageNet and MNIST, the suffixes ‘-N’ and ‘-S’ denote Natural and Synthetic distribution shifts, respectively. For BREEDS, ‘-S’ and ‘-N’ denote Same and Novel subpopulations. BREEDS-S consists of L17-S, NL26-S, E13-S, and E30-S; BREEDS-N consists of L17-N, NL26-N, E13-N, and E30-N. Dataset
Shift Type
Shift Dataset(s)
MNIST
–
QMNIST, USPS, SVHN
CIFAR-10
C10-N C10-S
CIFAR-10v2 (re-sampled natural test set) CIFAR-10-C (19 corruption types × 5 severity levels = 95 corruptions)
CIFAR-100
C100-N C100-S
CIFAR-100 test set (standard evaluation) CIFAR-100-C (19 corruption types × 5 severity levels = 95 corruptions)
IN-S
ImageNet-V2 (3 variants: Matched-Frequency, Threshold-0.7, Top-Images), ImageNet-Sketch ImageNet-C (19 corruption types × 5 severity levels = 95 corruptions)
Tiny-ImageNet
IN200-N IN200-S
Imagenet200-V2 (3 variants), Imagenet200-Sketch, ImageNet-Reality ImageNet200-C (19 corruption types × 5 severity levels = 95 corruptions)
FMoW
–
FMoW (OOD-val and OOD-test)
BREEDS
BREEDS-S BREEDS-N
IN-N and IN-S with same subpopulation hierarchies IN-N and IN-S with novel subpopulation hierarchies
ImageNet
IN-N
For MNIST, we consider QMNIST [59], USPS [18], and SVHN [34] as shifted variants. For CIFAR-10, we use CIFAR-10v2 [30] as a natural shift (C10-N) and CIFAR10-C [15] for synthetic corruptions (C10-S). For CIFAR-100, we use the standard test set as the natural shift benchmark (C100-N) and CIFAR-100-C [15] for synthetic corruptions (C100-S). For ImageNet, we assess natural shifts (IN-N) by ImageNet-V2 [39] and ImageNet-Sketch [54], and synthetic corruption (IN-S) by ImageNet-C [15]. For Tiny-ImageNet, the natural shifts (IN200-N) include ImageNet-Reality [16] and the corresponding 200 matching classes from ImageNetV2 and ImageNet-Sketch. The synthetic shift (IN200-S) utilizes the 200 matching classes extracted from ImageNetC. Note that the corruption benchmarks (CIFAR-10/100-C, ImageNet-C, ImageNet200-C) each contain 19 corruption types (e.g., Gaussian noise, motion blur, frost) with 5 severity levels per type, resulting in 95 distinct test conditions per dataset. FMoW naturally contains temporal and geographical distribution shifts and the out-of-distribution test data contain images from time periods and geographic regions unseen during training. For BREEDS datasets, we employ identical natural and synthetic shift protocols as ImageNet (V2 and C variants), models are evaluated on test sets containing either the training-time fine-grained categories (BREEDSS) or novel categories from the same coarse-grained class (BREEDS-N), which isolates the subpopulation shift effects.
D. Baselines We provide detailed formulations and implementation descriptions of all baselines compared in our experiments.
Importance Re-weighting (IM). The IM method estimates the target error as a re-weighted source error, where the weights are obtained as the ratio between the densities of target and source data across confidence bins. Following [3] this effectively corresponds to using a single slice in the classifier confidence space. Average Confidence (AC). THe AC baseline directly estimates the target error by computing the average of one minus the maximum softmax confidence over the unlabeled target samples. Difference of Confidence (DoC). It is also known as DOCFeat, which models the error as the difference between the source and target confidence distributions[11]. The formu lation is ε̂DoC = Ex∼DS I [arg maxj∈Y fθ (j | x) ̸= y] + Ex∼D T 1−maxj∈Y fθ (y | x) −Ex∼DS 1−maxj∈Y fθ (j | x) Generalized Disagreement Equality (GDE). It estimates the target error as the disagreement ratio between the predictions of two independently trained models fθ (x) and fθ′ (x) on the target data[19], which can be formulated as ε̂GDE =i h Ex∼DT I(arg maxj∈Y fθ (j | x) ̸= arg maxj∈Y fθ′ (j | x)) . Average Thresholded Confidence (ATC). ATC estimates target error by identifying a threshold t such that the fraction of source data points with scores below t matches the validation error on source data. The target error is then estimated as the proportion of target h examples falling i below this threshold: ε̂ATC = Ex∼DT I(s(fθ (x)) < t) , where s(·) denotes a scalar score function relating positively with
the performance of the model. As proposed in [10], there are two variants considered: (1) ATC-MC, which uses the maximum softmax confidence sMC = maxj∈Y fθ (j | x), and (2) ATC-NE, which uses the negative entropy score P sNE = − j∈Y fθ (j | x) log fθ (j | x).
Table 6. Ablation of thresholding strategy. The performance difference between the standard FRAP and FRAP w/o Thresholding denoted as FRAP (w/o thd) Dataset Family
Shift Type
Projection Normalization (ProjNorm). ProjNorm [61] originally proposed a parameter-space metric that quantifies the distributional shift between source and target domains. The original method does not directly estimate the target accuracy but instead demonstrates that the projection norm strongly correlates with the true target error. In our evaluation, since our goal is to directly predict model performance, we follow a practical implementation adapted from [31], which converts the original ProjNorm metric into an approximate accuracy estimator by comparing the output distributions between source and target data. This enables a fair, quantitative comparison under the same Mean Absolute Estimation Error (MAE) metric.
MNIST
Confidence Optimal Transport (COT). In [31], COT introduces an optimal-transport-based (OT) estimator that measures the Wasserstein distance between the empirical distribution of model confidence vectors on the unlabeled target set and the empirical source label distribution. Unlike confidence-based estimators (e.g., Average Confidence) which may underestimate error by selecting the pseudo-label distribution as reference, COT uses the source label distribution under the assumption that PT (⃗y ) ≈ PS (⃗y ). The paper further proposes COTT (COT with Thresholding), which learns a threshold on validation data and estimates error as the fraction of target samples whose per-sample transport costs exceed this threshold.
E. CWF on Additional Datasets To further validate the effectiveness of the confidenceweighted fusion (CWF), we conduct supplementary experiments on additional datasets including Tiny-ImageNet, Living-17, and Nonliving-26. We adopt the same evaluation protocol as described in Sec. 5.3 to assess how well the fused predictions align semantically with the ground truth using Semantic Alignment Score (SAS). Note that this hierarchybased evaluation is only applicable to datasets whose class labels conform to the WordNet taxonomy, which restricts our evaluation to the aforementioned datasets that satisfy this prerequisite. As shown in Figs. 5 to 7, the fused predictions on Living-17 and Nonliving-26 consistently exhibit markedly higher semantic consistency with the true labels than those of the base model. However, on Tiny-ImageNet, while the fused distribution substantially surpasses predictions of CLIP, it exhibits slightly degraded performance relative to the base model. This phenomenon suggests that poor performance of CLIP on this particular dataset adversely affects the fusion
FRAP
FRAP(w/o thd)
–
12.42±0.10
16.78±0.04
CIFAR-10
C10-N C10-S
2.14±0.02 3.24±0.05
6.13±0.04 7.26±0.05
CIFAR-100
C100-N C100-S
4.00±0.03 11.42±0.10
7.97±0.05 8.53±0.06
ImageNet
IN-N IN-S
2.16±0.03 4.76±0.01
12.31±0.07 9.16±0.06
Tiny-ImageNet
IN200-N IN200-S
7.46±0.04 9.98±0.04
14.7±0.05 12.37±0.06
FMoW
–
3.42±0.02
15.74±0.05
BREEDS-S
L17-S NL26-S E13-S E30-S
8.55±0.05 9.30±0.07 8.29±0.05 6.94±0.06
10.98±0.06 12.63±0.07 11.69±0.06 11.73±0.07
BREEDS-N
L17-N NL26-N E13-N E30-N
5.37±0.05 6.66±0.09 5.90±0.06 5.54±0.07
4.09±0.04 5.63±0.06 6.66±0.05 6.4±0.05
6.53
10.04
Average
outcome, where the predictions of base model are compromised by weaker contributions of CLIP. This observation highlights a critical caveat: while CLIP exhibits remarkable zero-shot generalization capabilities, its knowledge coverage remains bounded. In certain specialized domains, its predictions can fall considerably short of those from task-specific base models. Nevertheless, this finding underscores the merit of our fusion-based approach over directly adopting CLIP predictions as an approximation of the ideal distribution. The integration of CLIP represents a calculated trade-off: while it may introduce adverse effects in limited scenarios, the confidence-weighted fusion mechanism predominantly yields improved predictive distributions across the majority of settings by effectively leveraging the complementary strengths of both models.
F. Thresholding Strategy Ablation As detailed in Sec. 5.4, the thresholding strategy is introduced to address the inherent error of soft accuracy and the gap between the fused predictions and the true labels. This strategy transforms the continuous estimation problem into a more robust binary decision. By calibrating the threshold δ on the source validation set Ds , we effectively align the fraction of samples predicted as incorrect (i.e., Est(x) < δ)
Fusion vs. Base Model
Fusion vs. CLIP
Improvement: 82.7% Mean: 0.241
1.25
Fusion vs. Base Model
Improvement: 71.9% Mean: 0.184
0.75
0.5
0.50 0.0
0.25 0.00
0.5
0.25
Similarity Gap
Similarity Gap
1.00
0.50
0.6
1.0
250 500 750 1000 1250 1500 1750
Test Sample Index
0
Improvement: 55.7% Mean: 0.045
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0.2
0.2
0.4
0.4
0.6
1.0
0
Fusion vs. CLIP
Improvement: 86.2% Mean: 0.188
250 500 750 1000 1250 1500 1750
0.6
0
Test Sample Index
250 500 750 1000 1250 1500 1750
Test Sample Index
(a) Same shift
0
250 500 750 1000 1250 1500 1750
Test Sample Index
(b) Novel shift
Figure 5. Per-experiment Semantic Alignment Score (SAS) differences on Living-17. We compare the confidence-weighted fusion against the base model (left column) and CLIP (right column) under (a) natural shifts and (b) synthetic corruptions. The y-axis shows ∆SAS = SASfusion − SASbaseline . Positive values indicate improved semantic alignment (∆SAS > 0), while negative values indicate degradation. The fusion largely enhances alignment under both conditions. Fusion vs. Base Model
Fusion vs. CLIP
Improvement: 57.8% Mean: 0.043
1.25
0.50
0.0
0.25
0.5
Similarity Gap
Similarity Gap
0.5
0.75
0.00
0
250 500 750 1000 1250 1500 1750
Test Sample Index
0
Improvement: 87.1% Mean: 0.155 0.4
0.2
0.2
0.0
0.0
0.2
0.2
0.4
1.0
0.25
Fusion vs. CLIP
Improvement: 75.4% Mean: 0.076
0.4
1.0
1.00
Fusion vs. Base Model
0.6
Improvement: 97.5% Mean: 0.593
0.6
250 500 750 1000 1250 1500 1750
Test Sample Index
0.4
0
250 500 750 1000 1250 1500 1750
Test Sample Index
(a) Same shift
0
250 500 750 1000 1250 1500 1750
Test Sample Index
(b) Novel shift
Figure 6. Per-experiment Semantic Alignment Score (SAS) differences on Nonliving-26. (a) same shifts and (b) novel shifts is the same with the Living-17. The fusion consistently achieves better semantic alignment than both the base model and CLIP.
0.6
Similarity Gap
Fusion vs. CLIP
Improvement: 30.0% Mean: -0.081
Fusion vs. Base Model
Improvement: 90.0% Mean: 0.374
0.6 0.4
0.4 0.2
0.2
0.0 0.2
0.0
0.4
0.2
Fusion vs. CLIP
Improvement: 30.7% Mean: -0.047
1.5
Improvement: 99.2% Mean: 0.823
1.0
0.5 0.0
0.5
0.5
0.0
1.0
0.6
0
20
40
60
Test Sample Index
80
0
20
40
60
Test Sample Index
80
1.5 1.0
Similarity Gap
Fusion vs. Base Model
1.5
0.5
0
250 500 750 1000 1250 1500 1750
Test Sample Index
0
250 500 750 1000 1250 1500 1750
Test Sample Index
Figure 7. Per-experiment Semantic Alignment Score (SAS) differences on Tiny-ImageNet. (a) natural shifts and (b) synthetic corruptions is similar to the ImageNet. While the fused distribution on this dataset provides limited gains over the base model.
with the observed error rate of the base model on Ds . This process stabilizes the estimation by correcting the scale and offset between the raw prediction score and the actual error magnitude, relying on the relative ranking of scores rather than their absolute magnitudes.
where the label distribution is replaced by the fused prediction produced through test-time calibration (TTC) and confidence-weighted fusion (CWF). This ablation allows us to isolate the performance gain specifically attributable to the thresholding step.
To quantify the effect of the thresholding strategy, we perform an ablation study, which serves the formulation result in Eq. (A.7) directly as the performance estimation,
As illustrated in Tab. 6, ablation of the thresholding strategy results in a marked deterioration in estimation performance. Without thresholding, 15 out of 18 benchmark set-
Table 7. Comparison with extensive baselines. This table extends Tab. 4 with full results across all datasets. Background colors denote reference sources: CLIP , SigLIP , Random . Columns represent: (1) Base: reference pseudo-labels; (2) Fix: FRAP with fixed reference scaling; (3) Dyna: our full framework with TestTime Calibration (TTC). Random reports performance using a pathological reference via Dirichlet distribution. Data
Base
Fix
Dyna
Base
Fix
Dyna Random
MNIST 24.98.21 11.13.10 12.42.10 32.09.26 13.54.10 10.56.10 11.30.10 C10-N 5.00.15 8.48.03 2.14.02 2.01.00 6.29.02 7.54.03 11.05.04 C10-S 10.49.13 8.01.05 3.24.05 9.14.09 5.67.04 7.68.06 12.61.07 C100-N 9.07.08 10.90.05 4.00.03 4.35.01 4.65.05 10.22.04 12.71.06 C100-S 9.51.16 17.07.10 11.42.10 7.05.05 12.51.09 17.87.10 19.28.10 IN-S 7.08.29 3.11.02 4.76.01 4.02.02 3.57.02 3.74.02 2.99.02 IN-N 9.25.24 1.43.01 2.16.03 3.80.02 0.88.01 1.15.01 1.97.02 IN200-S 12.97.23 4.95.04 9.98.04 8.74.02 6.33.04 6.14.05 4.86.04 IN200-N 11.97.19 5.32.06 7.46.04 6.53.04 4.31.05 5.43.07 5.91.07 FMoW 40.86.10 3.39.03 3.42.02 41.27.07 2.20.02 2.70.03 2.48.03 L17-S 4.74.23 6.15.05 8.55.05 4.66.04 6.33.05 6.81.06 5.80.06 NL26-S 7.79.26 5.64.05 9.30.07 7.16.04 5.28.04 7.60.09 7.24.08 E13-S 10.39.19 4.93.05 8.29.05 12.38.05 4.55.04 5.78.07 5.75.06 E30-S 10.53.26 4.07.05 6.94.06 10.02.05 3.92.05 5.81.08 5.26.07 L17-N 1.82.22 5.70.05 5.37.05 1.65.02 5.26.04 6.74.07 10.20.07 NL26-N 0.85.18 5.74.05 6.66.09 1.02.01 4.91.04 8.48.10 9.04.09 E13-N 5.19.21 6.98.06 5.90.06 6.14.03 6.43.05 8.50.07 8.70.07 E30-N 3.19.22 6.01.05 5.54.07 3.77.02 5.08.05 9.05.09 8.70.07 Avg
10.32
6.61
6.53
9.21
5.65
7.32
8.10
tings exhibit a higher estimation error, and the average MAE rises substantially from 6.53 to 10.04. This pronounced drop underscores the effect of thresholding in our framework. The thresholding strategy effectively alleviates the accumulated error arising from the soft-to-hard score difference and the inherent limitation of the fused predictions.