Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction Jiazhen Huang 1 Zhiming Liu 1 Changhu Wang 2 Wei Ju 3 Ziyue Qiao 4 Xiao Luo 5
arXiv:2607.15851v1 [cs.CV] 17 Jul 2026
Abstract
on massive image-text datasets, these models have learned strong multimodal representation capabilities that align visual and textual concepts in a shared latent space. Therefore, VLMs generalize effectively to a wide range of downstream tasks and domains, from low-level image classification (Lu et al., 2022) to high-level visual question answering (Liu et al., 2023). This process requires few or even no training data. For example, CLIP (Radford et al., 2021) trains dual encoders through large-scale image-text contrastive learning. During inference, zero-shot predictions can be simply achieved by computing image-text embedding similarities.
A range of methods aim to enhance the performance of vision-language models (VLMs) at test time. Among them, transduction has emerged as a promising paradigm due to its strong compatibility and efficiency. However, realistic evaluations often involve highly imbalanced class distributions, which cause performance degradation or even collapse. In this work, we systematically revisit transduction from the perspective of penalized likelihood estimation (PLE), showing that PLE with a KL-divergence anchor term naturally yields an adaptive shrinkage behavior between prior anchors and empirical estimates. From this viewpoint, the brittleness of transductive methods can be attributed to the absence of anchoring mechanism and static modeling of the shrinkage strength. Therefore, we propose Mixture of Von Mises-Fisher Models with Dynamic Shrinkage (MOON). MOON is built upon a mixture of von Mises-Fisher distributions to model feature representations on the unit hypersphere. To handle imbalance, MOON dynamically adjusts the shrinkage strength using zero-shot priors at both instance and class levels. Thus, it suppresses unreliable assignments and prevents harmful updates from outlier classes, thereby mitigating negative transfer. MOON is model-agnostic, training-free, and requires no task-specific hyperparameter tuning. Extensive experiments further validate the advantage of MOON in both performance and efficiency.
Motivated by this, many methods aim to enhance the zeroshot predictive performance of VLMs at test time, where only unlabeled test samples are given. Among them, the most popular online test-time adaptation (TTA) (Döbler et al., 2024; Huang et al., 2026; Chen et al., 2026) methods adjust model behavior on-the-fly using current incoming data streams. In parallel, a closely related line of research, often referred to as transduction or transductive learning (Liu et al., 2020), focuses on exploiting the structure of available unlabeled data to perform joint inference over all test samples within a task1 . Specifically, transductive methods are typically achieved by performing soft probabilistic clustering at the embedding or logit level. Due to its “black-box” nature, which requires neither access to model internals nor expensive gradient backpropagation, transduction exhibits strong model-agnostic compatibility and high efficiency. Therefore, it has emerged as a particularly promising paradigm for realistic deployment. However, in realistic test-time scenarios, only small batches of unlabeled samples are available, where the underlying class distributions are often highly imbalanced. For instance, data streams may exhibit strong temporal correlations, or only a few classes may appear within a mini-batch. Nevertheless, most existing methods and benchmarks assume that class marginals are fixed and uniform. As discussed in prior studies (Veilleux et al., 2021), this often leads to degenerate predictions and severely limits their applicability. Our evaluations in Fig. 1 further demonstrate this brittleness that under realistic settings, many transductive and online TTA
1. Introduction Vision-language models (VLMs) have achieved remarkable success in the computer vision community. By pre-training 1 Tsinghua University, China 2 Fred Hutchinson Cancer Center, USA 3 Peking University, China 4 Great Bay University, China 5 University of Wisconsin-Madison, USA. Correspondence to: Wei Ju <[email protected]>, Ziyue Qiao <[email protected]>.
Proceedings of the 43 rd International Conference on Machine Learning, Seoul, South Korea. PMLR 306, 2026. Copyright 2026 by the author(s).
1 This notion is commonly contrasted with inductive learning, where predictions are independently made for each sample.
1
Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction Avg Very Low Keff
Medium Keff Bs=64
Bs=64
UCF101
Medium Keff Bs=1000
82.2%
69.5%
70.3%
+9.4% +7.7%
71.9% 67.0%
DTD
66.8%
45.3%
Caltech101
+2.2%
93.2%
94.4%
93.5%
72.2%
24.9% 24.6%
89.1%
-4.5%
CLIP
95.3%
TransCLIP
-8.7%
Pets
GDA-CLIP -16.1%
ADAPT StatA Moon(Ours)
(a) Batch adaptation with a limited number of effective classes.
CLIP
OGA
Aircraft
51.8%
55.4%
66.2% 85.6%
89.7%
94.8%
30.0%
51.6%
81.3%
76.4%
Flowers102
-3.9%
27.9%
24.7%
48.3% 65.6%
73.0%
77.0%
64.6%
62.5%
92.5% 70.7%
+0.1%
-22.5% -25.3%
66.6%
67.5%
43.5%
95.7%
66.5%
66.0%
65.2%
44.7%
+4.4%
77.2%
67.9%
68.9%
47.0%
SUN397
68.6%
69.8%
51.2%
+4.1%
+5.1%
-26.3% -30.0%
Imagenet
73.4%
74.6%
85.9%
59.0%
68.0%
EuroSAT
74.7% 89.3%
77.4% 95.8%
StanfordCars
Food101 ADAPT
StatA
Moon (Ours)
(b) Online adaptation under non-i.i.d. data streams.
Figure 1. Performance comparison on two realistic settings. Existing transductive or online TTA methods suffer from performance degradation or even collapse, while our proposed MOON consistently enhances VLM prediction and outperforms state-of-the-art baselines. Ours
methods suffer from performance degradation or even collapse. Statistically, the failures of transductive methods may arise as they implicitly treat prediction as a latent-variable estimation process. Under class imbalance, majority classes dominate the statistics, while estimates derived from limited, biased observations of minority classes become unreliable and increasingly deviate over iterations2 . This ultimately leads to negative transfer (Wang et al., 2019).
StatA CLIP ADAPT
EM-Dirichlet
TransCLIP GDA-CLIP ZLaP
(a) Anchor term weight 𝛼
To alleviate this issue, recent studies introduce explicit regularizations to penalize excessive deviation of empirical estimates, among which KL-divergence-based anchor terms have proven effective (Zanella et al., 2025). Based on these observations, transduction can be systematically revisited under a unified penalized likelihood estimation (PLE) formulation, where prior knowledge is incorporated as a statistical anchor. Specifically, we theoretically demonstrate that KL-anchored estimator naturally yields an adaptive shrinkage behavior, where class statistics are updated as a convex combination between prior anchors and empirical estimates. Hence, we attribute the brittleness of existing transductive methods to the following limitations: First, methods without anchoring mechanisms tend to overfit local statistics, which rapidly amplifies noisy assignments and leads to catastrophic collapse; Second, even anchor-based methods typically rely on a static modeling of shrinkage strength that implicitly assumes the reliability of statistics remains constant across samples and classes. As illustrated in Fig. 2(a), this design remains far from optimal in both accuracy and robustness, and inevitably requires task-specific hyperparameter tuning, which is impractical in practice.
(b) Accuracy-Runtime Trade-off
Figure 2. (a) Controlling shrinkage strength with anchor weight α of state-of-the-art method StatA. Such static modeling is suboptimal in accuracy and robustness. (b) Accuracy-Runtime Tradeoff. MOON enables effective and efficient adaptation.
using a mixture of von Mises-Fisher (vMF) distributions on the unit hypersphere. To robustly handle class imbalance, MOON dynamically adjusts the shrinkage strength using zeroshot priors at both instance and class levels. At the instance level, unreliable assignments are suppressed based on entropy, promoting a more robust label coverage. At the class level, harmful updates from outlier classes are identified and prevented, thus mitigating negative transfer. As a result, MOON enables a fine-grained, fully data-driven shrinkage, and can be seamlessly plugged into existing VLMs for enhancement. Our contribution can be concluded as follows: ❶ New Perspective with Theoretical Support. We systematically analyze the brittleness of existing methods under realistic class imbalance. By revisiting transduction from the perspective of penalized likelihood estimation (PLE), we theoretically prove that such estimators inherently exhibit adaptive shrinkage, which allows us to identify two limitations: absence of anchoring mechanism and static shrinkage strength modeling. ❷ Novel Methodology. We propose MOON, which is based on a mixture of von Mises-Fisher (vMF) distributions. It dynamically adjusts the shrinkage strength using zeroshot priors, which effectively suppresses unreliable assignments and prevents harmful updates from outlier classes,
Therefore, we propose MOON, a simple yet effective method for realistic test-time transduction. MOON follows the KLanchored PLE objective and models feature representations 2
This also applies to online TTA methods: although predictions are made sequentially, memory banks or distributional anchors are still influenced by previous statistics of the data stream.
2
Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction
mitigating negative transfer. ❸ Empirical Validation. We conduct extensive experiments across 11 datasets under two realistic settings, demonstrating that MOON is: (i) Effective: improving zero-shot CLIP by 13.2% across 10 scenarios on ImageNet, outperforming the strongest baseline by 8.8%; (ii) Efficient: Being training-free, processing thousands of samples within tens of milliseconds, which is merely 3.3% of CLIP’s inference latency; (iii) Practical: Operating under a model-agnostic black-box assumption and requiring no task-specific hyperparameter tuning, ensuring seamless deployment.
Consequently, both transductive and online TTA methods may suffer from performance degradation or even collapse, as also empirically demonstrated in prior works (Zhao et al., 2023; Veilleux et al., 2021). Recent solutions either mitigate sampling bias with memories (Gong et al., 2022) or introduce statistical regularization to stabilize estimates (Zanella et al., 2025). Our MOON aligns with the latter by adopting a KL-anchored PLE framework with dynamic shrinkage.
3. Revisiting Test-Time Transduction Problem definition. Consider a batch of N test samples {xi }N i=1 , with the label space consisting of K candidate classes. Let θv (·) and θt (·) denote the visual and textual encoders of a pre-trained VLM, respectively. For each class k, we obtain its textual embedding tk = θt (ck ) ∈ Rd with a prompt ck (e.g., “a photo of a [classname]”). Similarly, the visual feature embedding is extracted as fi = θv (xi ) ∈ Rd . After ℓ2 -normalized onto the unit hypersphere Sd−1 , zeroshot predictions are computed via cosine similarity:
2. Related Work Enhancing VLMs at Test-Time. A growing body of work focuses on adapting VLMs at test time to enhance their performance on downstream tasks, where only unlabeled target samples are available. Among them, the most popular testtime adaptation (TTA) methods can be broadly divided into two categories. The first focuses on updating model parameters online via lightweight fine-tuning, such as prompt tuning (Shu et al., 2022; Feng et al., 2023) or adapters (Abdul Samadh et al., 2023). This often requires expensive data augmentation and gradient backpropagation. The second avoids training and instead directly adjusts model outputs by maintaining caches (Zhang et al., 2024a), memories (Zhang et al., 2024b), or distribution modeling (Han et al., 2024) over historical data streams. Our work primarily relates to this category and refers to these methods as online TTA. In parallel, the concept of transduction was originally explored in few-shot learning (Martin et al., 2022), where it aims to exploit the structure of available data to perform joint inference over test samples. When extended to zero-shot learning, this paradigm can be viewed as a subclass within a broader TTA framework. Recent works like TransCLIP (Zanella et al., 2024) and ZLaP (Kalantidis et al., 2024) have investigated transduction for VLMs, while StatA (Zanella et al., 2025) further discusses it under the test-time class imbalance. Despite making progress, several limitations still bottleneck their performance. This motivates our MOON.
ŷi = {ŷi,k }K k=1 ∈ ∆K ,
exp(fi⊤ tk /τ ) ŷi,k = P , ⊤ j exp(fi tj /τ )
(1)
where ∆ denotes the probability simplex, and τ is a fixed temperature coefficient from pre-training. Realistic imbalanced settings. Following StatA (Zanella et al., 2025), we consider two realistic settings: (i) Batch adaptation: each batch contains a limited number of effective classes Keff (1 ≤ Keff ≤ min{N, K}), where batches are processed independently. (ii) Online adaptation: test samples arrive as a non-i.i.d. data stream, whose temporal correlation is controlled by a Dirichlet parameter ξ. Here, historical batch information is accessible. For details on the sampling for the imbalanced data, please refer to App. C.4. Penalized likelihood estimation. Recent transductive methods can be broadly conceptualized as a family of soft probabilistic clustering algorithms, which aim to infer latent class assignments and class-conditional distributions over the unlabeled test set. We revisit this process within a penalized likelihood estimation (PLE) formulation that jointly estimates the following variables: (i) Assignment vectors zi = {zi,k }K k=1 ∈ ∆K , representing the latent posterior class probability within the probability simplex (initialized from ŷi ). (ii) Mixture models M = {Mk }K k=1 , where each component Mk models the feature distribution of class k with a set of statistical parameters (e.g., mean and covariance). The general optimization objective is given by:
Imbalanced Learning in Realistic Scenarios. Most existing TTA methods and benchmarks assume perfectly classbalanced tasks at inference, i.e., the marginal class probabilities are treated as uniform. In contrast, realistic deployment scenarios often exhibit highly imbalanced class distributions, such as sparse, long-tailed, or non-i.i.d. batches (Ochal et al., 2023). In such contexts, the inductive biases of standard TTA methods can be harmful. First, limited and biased statistics might lead models to overfit to locally dominant distributions, resulting in negative transfer. Second, commonly adopted tricks like marginal entropy minimization (Wang et al., 2020) also become counter-productive as they force the model to align with a mismatched uniform prior.
arg min LPLE = arg min z,M
z,M
−
N X z⊤ i log pi + R(z) + αR(M). i=1
(2) 3
Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction
Limitation 1: no anchor. Notably, in the absence of the anchor term (α = 0), we have βk ≡ 1, and Eq. (5) degenerates to standard MLE. Under realistic class imbalance, this causes estimation overfitting to locally biased statistics dominated by majority classes, which can rapidly amplify errors over iterations and lead to catastrophic collapse.
Here, the first term represents the standard negative loglikelihood (NLL), with pi denoting class-conditional likelihoods under M. The terms R(z) and R(M) serve as penalization regularizers for assignments and distribution parameters, respectively. Specifically, R(z) is typically introduced to mitigate the inherent biases of unsupervised clustering, e.g., by encouraging smoothness or consistency with priors. R(M) is the key to prevent the model from overfitting to local statistics in realistic class-imbalanced scenarios, where α is a hyperparameter. This is achieved by penalizing the deviation between empirical estimate and the zero-shot prior anchor M′ via a KL divergence: R(M) = KL(M′ ∥M).
Limitation 2: bounded but static shrinkage. When α > 0, Eq. (5) further implies ∥∇A(ηk⋆ ) − µ′k ∥ = βk ∥µ̂k − µ′k ∥, linearly bounding the deviation by βk . In particular, if nk = 0 (outlier classes), the update stays exactly at its anchor, i.e., ∇A(ηk⋆ ) = µ′k , preventing any harmful deviation. However, we could find that these types of anchor-based methods still employ a static anchor strength α, implicitly assuming equal reliability of statistics across instances and classes. Therefore, their performance and robustness may remain suboptimal (as shown in Fig. 2(a)), which inspires our MOON. Detailed proofs are provided in App. D.
Adaptive anchor shrinkage. The effectiveness of the PLE formulation in realistic class imbalance largely hinges on the KL-based distribution anchor term R(M). We theoretically demonstrate that KL-anchored PLE naturally yields an adaptive shrinkage behavior, where statistics are encouraged to update between empirical estimates and prior anchors. While recent works are implemented under a standard Gaussian assumption by default, this behavior naturally generalizes to the entire exponential family of distributions.
4. Our Proposed MOON Since Eq. (5) applies to the entire exponential family, we propose to adopt a mixture of von Mises-Fisher (vMF) (Gopal & Yang, 2014; Hasnat et al., 2017; Govindarajan et al., 2024) distributions for modeling as normalized VLM embeddings are intrinsically constrained to the unit hypersphere. This distribution is commonly regarded as the natural generalization of Gaussian distribution onto the sphere (Martin et al., 2024). Formally, for a d-dimensional unit vector fi ∈ Sd−1 , the probability density function of a vMF component Vk (µk , κk ) is defined as:
Formally, consider a K-class latent-variable mixture model with soft assignments {zi,k }N i=1 and class-conditional densities from a regular minimal exponential family, i.e., p(x | η) = h(x) exp η ⊤ T (x) − A(η) , (3) where η is the natural parameter, T (x) represents the suf3 ficient statistic, A(η) is the log-partition function , and P h(x) denotes the base measure. Let n = z k i i,k and P Sk = i zi,k T (xi ) denote the class-wise soft count and sufficient-statistic sum. Given a fixed anchor distribution qk (x) = p(x | ηk′ ) with mean parameter µ′k = ∇A(ηk′ ), the KL-anchored PLE optimizes during parameter update:
⊤ pvMF i,k = p(fi ; µk , κk ) ∝ Cd (κk ) exp(κk µk fi ),
(6)
where µk ∈ Sd−1 denotes the mean direction vector, and κk ≥ 0 is a scalar concentration parameter measuring the d/2−1
κ
isotropic precision. Cd (κk ) = (2π)d/2kI is the norN X K K d/2−1 (κk ) X X min − zi,k log p(xi | ηk )+α KL qk ∥ p(· | ηk ) , α > 0. malization constant, derived from the order-ν modified {ηk } i=1 k=1 k=1 Bessel function of the first kind Iν (·). On the basis of (4) this, the KL divergence between Vk and anchor Vk′ takes: Then the unique minimizer ηk⋆ satisfies the closed-form C (κ′ ) ′ KL (Vk′ ∥Vk ) = log Cdd (κkk ) + κ′k Ad (κ′k ) − κk Ad (κ′k )µ⊤ k µk shrinkage in the mean-parameter space: Id/2 (κ) (details in App. E.2), where Ad (κ) = Id/2−1 (κ) represents Sk + αµ′k ⋆ ′ ∇A(ηk ) = = βk µ̂k + (1 − βk )µk , (5) the Bessel function ratio. We initialize the anchor distribunk + α tion Vk′ with zero-shot priors leveraged from text: Sk where µ̂k = nk is the empirical estimate when nk > 0, P k zi,k ∥fi − µ′k ∥2 βk = nkn+α ∈ [0, 1] denotes the shrinkage strength. Eq. (5) µ′k = tk , Ad (κ′k ) = 1 − i P . (7) 2 i zi,k reveals that the update is a data-driven convex combination between the empirical estimate and the prior anchor, with βk , The derivation of Ad (κ′k ) is provided in App. F. Building controlled by soft count nk and anchor weight α, adapting upon the above formulation, we arrive at the final objective: automatically to the amount of evidence available for each XN vMF class: classes with abundant support approach standard LPLE (z; µ, κ) = γ − z⊤ ) + R(z) i log(pi i=1 maximum likelihood estimation (MLE), while rare classes XK (8) +α KL Vk′ ∥Vk , remain strongly regularized by the anchor. k=1 3
where
This is different from the Bessel function ratio Ad (·) below.
4
R(z) = −
X i,j
ωij z⊤ i zj +
XN i=1
KL(zi ∥ŷi ).
Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction
For R(z), we choose a widely-adopted combination of a Laplacian regularizer and a text-supervision term (Zanella et al., 2024; 2025), where ωij = fi⊤ fj denotes feature affinity. The former performs label propagation among nearby samples to encourage smooth assignments, while the latter penalizes deviations from zero-shot predictions. Moreover, we introduce two weights at instance and class level, γ and α, enabling dynamic adjustment of shrinkage strength. Different from Eq. (2), both weights are driven by priors.
mismatch, which should be suppressed during adaptation. In PDA, such mismatch is typically quantified with zero-shot prediction confidence, since classes with higher confidence are more likely and frequent to be present in the target domain. Leveraging this insight, we replace the fixed scalar α with class-level dynamic weights α = {αk }K k=1 . Intuitively, highly confident classes should encourage the parameters µk and κk to align more closely to empirical k estimates, pushing the strength βk = nkn+α towards 1. k Therefore, αk should be negatively correlated with class confidence. We define αk to be inversely related to confidence, as this form is widely used in statistical learning to impose regularization on less reliable signals (Zou, 2006):
4.1. Dynamic Shrinkage for Realistic Class Imbalance Instance-level adjustment. The first two terms in Eq. (8) actually form a standard MLE objective, which typically treats all test samples equally. However, in realistic scenarios, certain zero-shot predictions may be inherently noisy; estimation biases may also accumulate and propagate over iterations. A natural idea is to employ predictive entropy as a metric for reliability, as it is widely adopted in TTA for model optimization or memory updates (Wang et al., 2020; Karmanov et al., 2024). Therefore, we introduce an entropy-based weight γi ∈ [0, 1] to dynamically re-weight the contribution of each sample to the MLE objective: γi = 1 −
H(ŷi ) , log K
1 , (10) λk where λk denotes the k-th class confidence derived from zero-shot priors. PDA methods often directly estimate λk from average confidence. In test-time settings, however, such a design is insufficient, as the effective label set varies across batches. On the one hand, those rare but effective classes, occurring infrequently yet consistently in the data streams, might be confused with truly outlier classes and instead suppress positive transfer. On the other hand, we don’t want to lose the generality under a distribution closer to uniform. For balance, λk is defined as the geometric mean of the average and the maximum confidence: αk =
(9)
PK
where H(ŷi ) = − k=1 ŷi,k log ŷi,k denotes entropy, and log K serves as the normalization factor. Through this mechanism, certain predictions are encouraged, while uncertain or ambiguous ones are suppressed. As explicitly shown in Eq. (14), it serves as a coefficient for assignments zi and filters out unreliable samples during the parameter estimation of µk and κk . In implementation, we update γi with i) current assignments zi for stability, i.e., γi = 1 − H(z log K .
r λk =
1 XN ŷi,k ⊙ max ŷi,k . i=1 i N
(11)
This mildly sacrifices accuracy under severe class imbalance, but yields a more general solution across broader scenarios. 4.2. Optimization Algorithm
Class-level adjustment. Class imbalance inherently induces class sparsity, such as Keff ≪ K or ξ → 0. More importantly, it’s impossible to identify which classes are effective (i.e., present) within batch, as labels are unavailable at test time. This makes transduction particularly vulnerable to negative transfer from outlier (i.e., absent) classes. Although the distribution anchor in Eq. (8) alleviates this issue by inducing an adaptive shrinkage behavior, it treats all classes equally and lacks dynamic, fine-grained shrinkage strength modeling. Moreover, it operates with a single scalar hyperparameter α, which requires task-specific tuning.
Since the proposed PLE objective jointly involves the assignments z and mixture parameters {Vk (µk , κk )}K k=1 , we adopt an efficient optimization algorithm following recent works (Zanella et al., 2024). The algorithm follows the Block Successive Minimization (BSUM) framework (Razaviyayn et al., 2013), which alternately updates two blocks of variables via iterative block-coordinate descent on surrogate objectives. This algorithm is also theoretically guaranteed to converge, as shown in App. A. Given that both γ and α are non-negative, all terms in Eq. (8) except the Laplacian regularizer are convex with respect to each block.
To address this limitation, Partial Domain Adaptation (PDA) (Cao et al., 2018) has provided successful experiences that we can learn from. PDA studies settings in which the target label space forms a subset of the source label space. This also works for our test-time settings, as the effective class set of a given test batch is also a subset of VLM’s predefined candidate class set. Consequently, the absence of classes can be interpreted as a form of source-target label space
Linear approximation w.r.t assignmentsP z. Due to the presence of concave Laplacian regularizer i,j ωij z⊤ i zj , a closed-form update for z cannot be obtained directly. Therefore, we construct a linear upper bound by replacing this term with its first-order Taylor expansion at current iteration, P ⊤ P (t) (t) i.e., − i zi , where zj denotes the assignj ωij zj ment obtained at iteration t. By minimizing the constructed 5
Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction
P (t) ŷi ⊙ exp(log pvMF + j ωij zj ) i . P (t) (ŷi ⊙ exp(log pvMF + j ωij zj ))⊤ 1K i
70
(12)
60
CLIP TransCLIP ADAPT StatA Ours
50
Detailed derivations are provided in App. G.1. In implementation, we omit the inner-loop optimization required in previous works and perform a single pass per iteration for efficiency. Note that γ does not appear in Eq. (12) as it could be canceled out during the derivation of zi .
40
1 2 3 4 5 6 7 8 9 10
Iteration
49 48 47 46 45 44 43
DTD
CLIP TransCLIP ADAPT StatA Ours
1 2 3 4 5 6 7 8 9 10
Iteration
Figure 3. Convergence analysis on ImageNet and DTD. We demonstrate performance curves over iterations for each method.
5. Experiments
Closed-form update w.r.t parameters µ and κ. When fixing z, Eq. (8) becomes strictly convex with respect to the mixture parameters µ and κ. Therefore, we can derive closed-form updates by setting partial derivatives to zero: P γi zi,k fi + αk Ad (κ′k )µ′k µk = Pi , ∥ i γi zi,k fi + αk Ad (κ′k )µ′k ∥ P ∥ i γi zi,k fi + αk Ad (κ′k )µ′k ∥ P Ad (κk ) = . i γi zi,k + αk
Imagenet
Accuracy(%)
(t+1) zi =
80
Accuracy(%)
approximate surrogate objective, we have:
We evaluate our method in several scenarios under two realistic settings, as defined in Sec. 3. We report the Top-1 accuracy across 11 public fine-grained classification datasets, and adopt CLIP ViT-B/16 as our default VLM backbone. Please see App. C for details on datasets, baselines, prompt templates, and other experimental specifics.
(13)
5.1. Main Results
Batch adaptation. We first report the results under batch adaptation in Tab. 1(a) and (b), with batch sizes of 64 and 1,000, respectively. The results show that existing transductive methods generally suffer from severe performance degradation under realistic class-imbalance, and most βk v k + (1 − βk )µ′k µk = , Ad (κk ) = ∥βk v k + (1 − βk )µ′k ∥, of them even collapse and underperform zero-shot CLIP. ∥βk v k + (1 − βk )µ′k ∥ (14) While StatA mitigates this issue by introducing anchor term PN PN γi,k zi,k fi i=1 γi,k zi,k P R(M), its performance remains suboptimal. In contrast, our where v k = Pi=1 and β = . k N N i=1 γi,k zi,k i=1 γi,k zi,k +αk MOON consistently achieves the best average performance This offers an intuitive interpretation of the anchor shrinkage across all scenarios, effectively enhancing VLM predictions. as in Eq. (5). Here, v k represents empirical estimates from ′ Notably, the performance gains of MOON become more prostandard MLE, and µk serves as prior anchor. OurP proposed Keff nounced as class imbalance becomes more severe ( min(N,K) adjustments are seamlessly integrated here: nk = i γi zi,k P decreases). Moreover, MOON delivers the most significant replaces soft count i zi,k , ensuring that noisy predictions gains on challenging large-scale datasets such as ImageNet, are suppressed. αk further penalizes harmful deviations: for highlighting its superiority in practical applications. effective classes, αk → 1 while nk increases, allowing the model to learn more from data; for outlier classes, αk dominates βk , forcing the updates to shrink towards the anchor, Online adaptation. We further evaluate methods under thereby mitigating negative transfer. online adaptation. As shown in Tab. 2, most online TTA methods remain relatively stable across different correlation strengths, without exhibiting performance degradation. NevOverall procedure. The overall procedure of MOON is ertheless, MOON still achieves state-of-the-art performance summarized in App. A. The initializations and updates generally. We observe a slight drop only in the Low scenario, mentioned above directly yield the Bessel function ratio where the class distribution becomes closer to uniform. This Ad (κk ), which corresponds to the mean resultant length of can be attributed to the inherent bias of our α, as it is dethe vMF distribution r̄k . We then employ the well-known signed to favor sparse effective class sets. Similarly, MOON approximation (Banerjee et al., 2005) to estimate κk : brings better improvements in scenarios with stronger correlations. For example, MOON outperforms StatA by 10.7% d r̄k − r̄k3 κk ≈ , r̄k ≜ Ad (κk ). (15) 2 on ImageNet in the Separate scenario. 1 − r̄k
As shown in Sec. 3, we can rewrite the above updates in a more intuitive form. Under the mild assumption Ad (κ′k ) ≈ 1, Eq. (13) is equivalent to (proof in App. H):
Note that the parameter estimation of vMF mixtures is simpler and more computationally efficient than GMMs, as it uses fewer parameters and avoids the expensive quadratic forms and inversions of Rd×d covariance matrices.
5.2. Efficiency Analysis Runtime. Tab. 3 reports the runtime per batch on the ImageNet dataset. We observe that the CLIP inference, in6
Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction Table 1. Main results for batch adaptation, averaged over 1,000 runs. The best and second-best results are marked in bold and underlined, respectively. We report three scenarios for each batch size 64 and 1,000, with varying range of effective classes Keff . Subscript green indicates improvement, red indicates decline, and gray indicates no change compared with zero-shot performance.
01
2
D
U CF 10
48.3 46.9−1.4
65.6 68.0+2.4
85.9 87.2+1.3
89.1 89.4+0.3
70.7 71.7+1.0
93.2 94.0+0.8
43.5 44.4+0.9
67.5 69.0+1.5
65.2 66.6+1.3
Very Low (1–4)
Dirichlet ZLaP GDA-CLIP TransCLIP ADAPT StatA MOON
79.2+12.6 14.5−52.1 20.5−46.1 21.6−45.0 60.8−5.8 72.9+6.3 82.8+16.2
75.7+13.2 13.0−49.5 19.8−42.7 21.1−41.4 56.0−6.5 66.0+3.5 77.2+14.7
28.2+3.5 8.4−16.3 10.3−14.4 11.6−13.1 21.4−3.3 29.3+4.6 32.0+7.3
47.2−1.1 36.6−11.7 39.9−8.4 45.1−3.2 45.9−2.4 56.8+8.5 53.7+5.4
68.2+2.6 23.7−41.9 28.2−37.4 34.7−30.9 59.7−5.9 76.2+10.6 78.6+13.0
88.1+2.2 31.9−54.0 51.4−34.5 59.2−26.7 81.4−4.5 90.3+4.4 96.4+10.5
87.5−1.6 57.0−32.1 60.4−28.7 72.4−16.7 84.7−4.4 95.5+6.4 96.0+6.9
71.2+0.5 22.4−48.3 29.6−41.1 36.4−34.3 66.8−3.9 77.6+6.9 77.7+7.0
88.8−4.4 52.4−40.8 52.9−40.3 62.3−30.9 90.4−2.8 93.0−0.2 94.9+1.7
50.3+6.8 13.0−30.5 39.9−3.6 26.1−17.4 39.2−4.3 46.1+2.6 55.6+12.1
69.0+1.5 29.2−38.3 35.2−32.3 37.7−29.8 61.4−6.1 70.2+2.7 75.7+8.2
68.5+3.3 27.5−37.8 35.3−30.0 38.9−26.3 60.7−4.5 70.4+5.1 74.6+9.4
Low (2–10)
Dirichlet ZLaP GDA-CLIP TransCLIP ADAPT StatA MOON
80.1+13.5 19.1−47.5 18.6−48.0 20.3−46.3 65.3−1.3 72.8+6.2 83.8+17.2
78.0+15.5 19.0−43.5 19.4−43.1 22.4−40.1 60.7−1.8 66.9+4.4 78.0+15.5
28.1+3.4 12.0−12.7 12.6−12.1 14.3−10.4 23.5−1.2 27.7+3.0 29.8+5.1
43.5−4.8 46.4−1.9 50.3+2.0 53.9+5.6 51.5+3.2 51.3+3.0 48.3±0.0
71.5+5.9 27.9−37.7 25.2−40.4 30.8−34.8 62.3−3.3 73.5+7.9 76.7+11.1
92.3+6.4 43.5−42.4 49.7−36.2 55.6−30.3 84.5−1.4 89.5+3.6 95.5+9.6
92.7+3.6 66.6−22.5 60.4−28.7 69.4−19.7 87.5−1.6 93.7+4.6 94.6+5.5
74.7+4.0 31.3−39.4 33.1−37.6 40.9−29.8 68.8−1.9 76.6+5.9 77.3+6.6
93.0−0.2 60.8−32.4 55.2−38.0 64.6−28.6 91.0−2.2 93.6+0.4 95.3+2.1
48.9+5.4 22.4−21.1 28.4−15.1 31.6−11.9 41.8−1.7 46.9+3.4 50.9+7.4
70.9+3.4 38.7−28.8 37.2−30.3 40.9−26.6 63.8−3.7 69.6+2.1 74.2+6.7
70.3+5.1 35.2−30.0 35.5−29.8 40.4−24.8 63.7−1.5 69.3+4.1 73.1+7.9
Medium (5–25)
Dirichlet ZLaP GDA-CLIP TransCLIP ADAPT StatA MOON
77.7+11.1 29.0−37.6 19.2−47.4 15.5−51.1 66.8+0.2 70.7+4.1 79.5+12.9
72.9+10.4 27.9−34.6 21.4−41.1 22.8−39.7 61.7−0.8 65.3+2.8 72.6+10.1
26.1+1.4 16.5−8.2 15.8−8.9 17.0−7.7 25.0+0.3 26.0+1.3 26.0+1.3
38.6−9.7 49.0+0.7 56.2+7.9 58.2+9.9 52.8+4.5 45.0−3.3 42.9−5.4
71.6+6.0 36.0−29.6 27.4−38.2 32.9−32.7 65.4−0.2 71.1+5.5 73.6+8.0
90.8+4.9 59.1−26.8 52.9−33.0 56.3−29.6 85.9±0.0 88.2+2.3 92.5+6.6
88.4−0.7 76.4−12.7 68.7−20.4 72.6−16.5 88.7−0.4 90.8+1.7 90.7+1.6
71.5+0.8 42.9−27.8 40.1−30.6 45.0−25.7 69.7−1.0 73.7+3.0 74.4+3.7
93.7+0.5 72.0−21.2 59.2−34.0 65.6−27.6 92.2−1.0 93.9+0.7 94.6+1.4
42.9−0.6 32.0−11.5 35.2−8.3 37.5−6.0 43.5±0.0 47.5+4.0 44.7+1.2
67.8+0.3 50.3−17.2 43.7−23.8 46.5−21.0 66.7−0.8 69.1+1.6 71.5+4.0
67.5+2.2 44.7−20.6 40.0−25.3 42.7−22.5 65.3+0.1 67.4+2.2 69.4+4.1
TD
Pe ts
1
Ca lte
ch 1
10 Fl ow
24.7 27.4+2.7
er s
d1 Fo o
01
dC or St
62.5 64.8+2.3
an f
Eu r
AT
ra ft irc A
66.6 69.3+2.7
oS
7 39 SU
Avg.
CLIP MTA
Keff
N
eN
Method
Im ag
et
ar
s
(a) Setting where the batch size is 64: Very Low (1–4 Keff ), Low (2–10), and Medium (5–25).
10 1 Fo od
ow
te
Pe
Fl
Ca l
D TD
U CF 10 1
24.7 27.4+2.7
48.3 46.9−1.4
65.6 68.0+2.4
85.9 87.2+1.3
89.1 89.4+0.3
70.7 71.7+1.0
93.2 94.0+0.8
43.5 44.4+0.9
67.5 69.0+1.5
65.2 66.6+1.3
Medium (5–25)
Dirichlet ZLaP GDA-CLIP TransCLIP ADAPT StatA MOON
60.9−5.7 16.6−50.0 32.3−34.3 39.9−26.7 55.8−10.8 70.8+4.2 81.6+15.0
75.4+12.9 20.1−42.4 32.5−30.0 42.7−19.8 52.8−9.7 64.5+2.0 75.5+13.0
26.7+2.0 16.4−8.3 19.0−5.7 22.0−2.7 23.3−1.4 28.4+3.7 29.6+4.9
38.8−9.5 49.0+0.7 61.0+12.7 63.1+14.8 63.5+15.2 60.4+12.1 58.9+10.6
74.1+8.5 32.2−33.4 39.5−26.1 49.9−15.7 55.9−9.7 74.0+8.4 76.1+10.5
76.2−9.7 55.5−30.4 68.1−17.8 80.6−5.3 77.4−8.5 87.5+1.6 93.1+7.2
91.0+1.9 76.4−12.7 79.6−9.5 87.9−1.2 85.2−3.9 93.1+4.0 92.4+3.3
71.6+0.9 40.6−30.1 50.3−20.4 58.7−12.0 67.1−3.6 77.5+6.8 77.3+6.6
92.4−0.8 67.7−25.5 69.1−24.1 79.1−14.1 89.0−4.2 92.8−0.4 94.9+1.7
36.2−7.3 34.2−9.3 39.2−4.3 42.9−0.6 43.0−0.5 47.1+3.6 49.0+5.5
65.4−2.1 48.1−19.4 49.5−18.0 55.0−12.5 62.3−5.2 70.2+2.7 73.6+6.1
64.4−0.8 41.5−23.7 49.1−16.1 56.5−8.7 61.4−3.9 69.7+4.4 72.9+7.7
High (25–50)
Dirichlet ZLaP GDA-CLIP TransCLIP ADAPT StatA MOON
17.3−49.3 23.8−42.8 37.8−28.8 43.9−22.7 60.4−6.2 71.9+5.3 81.8+15.2
37.3−25.2 32.2−30.3 41.5−21.0 49.6−12.9 57.9−4.6 66.4+3.9 74.9+12.4
21.0−3.7 22.2−2.5 23.8−0.9 24.8+0.1 24.9+0.2 25.9+1.2 25.5+0.8
37.9−10.4 49.3+1.0 62.4+14.1 64.0+15.7 64.0+15.7 60.7+12.4 59.3+11.0
65.4−0.2 45.4−20.2 49.2−16.4 57.3−8.3 60.7−4.9 73.6+8.0 74.7+9.1
46.3−39.6 74.9−11.0 76.1−9.8 83.0−2.9 82.1−3.8 88.0+2.1 91.0+5.1
81.3−7.8 86.5−2.6 90.9+1.8 91.4+2.3 91.8+2.7 91.4+2.3 89.7+0.6
46.3−24.4 56.2−14.5 64.7−6.0 69.1−1.6 70.2−0.5 76.7+6.0 75.2+4.5
80.5−12.7 79.7−13.5 77.9−15.3 85.5−7.7 90.2−3.0 93.2±0.0 94.8+1.6
21.1−22.4 43.6+0.1 47.4+3.9 47.5+4.0 46.9+3.4 47.9+4.4 45.1+1.6
43.6−23.9 60.8−6.7 61.7−5.8 65.4−2.1 67.0−0.5 71.5+4.0 71.8+4.3
45.3−20.0 52.2−13.0 57.6−7.6 62.0−3.3 65.1−0.1 69.8+4.5 71.3+6.0
Very High (50–100)
Dirichlet ZLaP GDA-CLIP TransCLIP ADAPT StatA MOON
10.8−55.8 32.7−33.9 41.7−24.9 44.5−22.1 64.7−1.9 71.8+5.2 80.4+13.8
15.7−46.8 44.0−18.5 48.5−14.0 53.0−9.5 61.9−0.6 67.1+4.6 71.9+9.4
17.5−7.2 25.4+0.7 26.3+1.6 25.6+0.9 25.8+1.1 23.9−0.8 23.4−1.3
37.8−10.5 49.3+1.0 62.4+14.1 64.1+15.8 64.0+15.7 60.7+12.4 59.3+11.0
51.2−14.4 55.2−10.4 56.5−9.1 60.9−4.7 64.9−0.7 70.2+4.6 70.7+5.1
29.1−56.8 83.3−2.6 82.8−3.1 85.2−0.7 85.5−0.4 87.1+1.2 87.8+1.9
79.3−9.8 87.3−1.8 92.3+3.2 91.9+2.8 92.6+3.5 91.1+2.0 89.3+0.2
24.3−46.4 64.8−5.9 73.1+2.4 74.3+3.6 73.2+2.5 74.3+3.6 72.3+1.6
59.1−34.1 87.9−5.3 86.5−6.7 90.5−2.7 92.5−0.7 93.7+0.5 93.3+0.1
19.0−24.5 45.2+1.7 49.2+5.7 48.1+4.6 48.0+4.5 48.0+4.5 44.3+0.8
26.1−41.4 67.8+0.3 70.5+3.0 70.7+3.2 70.3+2.8 70.7+3.2 68.9+1.4
33.6−31.6 58.4−6.8 62.7−2.5 64.4−0.8 67.6+2.3 69.0+3.7 69.2+4.0
cluding both visual and textual encoding, dominates the total computational cost. Considering the net runtime of methods, our MOON highlights its exceptional efficiency. Despite requiring iterative optimization, MOON is still twice as fast as the single-pass ADAPT. Moreover, this efficiency advantage becomes increasingly pronounced as the batch size grows. This empirically validates a significant advantage of vMF mixtures that has been largely overlooked in prior works. We further provide a complexity analysis in App. A.
ts
er
ch
an f St
s1
10 1
02
dC ar s or
Eu ro
SA T
t A irc
62.5 64.8+2.3
ra f
7
66.6 69.3+2.7
Keff
N 39 SU
Avg.
CLIP MTA
ag
Method
Im
eN et
(b) Setting where the batch size is 1,000: Medium (5–25 Keff ), High (25–50), and Very High (50-100).
a single iteration could yield competitive performance. 5.3. Ablation Studies Components. We study the effect of key components in Tab. 4. Obviously, iteratively updating parameters (µ, κ) improves performance, as it allows the class-conditional distributions to progressively adapt to the underlying latent structure of data. By dynamically adjusting shrinkage strength, our α robustly mitigates negative transfer from outlier classes, and its benefit scales up as the scenario becomes more imbalanced. Surprisingly, the effect of γ is marginal when batch size is small. A possible explanation is that the estimation error is dominated by high statistical variance and class sparsity in small batches; whereas in large batches, sufficient dense samples provide enough stable statistics for γ to filter out noise. Overall, simply incorporating zero-shot
Convergence. Fig. 3 illustrates the convergence curves of four transductive methods on ImageNet and DTD. We find that our MOON converges rapidly within just a few iterations, maintaining stable and consistent improvements. In contrast, methods without anchoring mechanism (e.g., TransCLIP) tend to deviate from global, reliable estimates as the iteration proceeds, leading to collapse. Notably, even 7
Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction
Ca lte c
D
U
24.7 27.4+2.7
48.3 46.9−1.4
65.6 68.0+2.4
85.9 87.2+1.3
89.1 89.4+0.3
70.7 71.7+1.0
93.2 94.0+0.8
43.5 44.4+0.9
67.5 69.0+1.5
65.2 66.6+1.3
Low (ξ = 0.1)
TENT TDA DMN OGA ADAPT StatA MOON
66.6±0.0 68.3+1.7 68.0+1.4 68.8+2.2 69.2+2.6 66.2−0.4 67.0+0.4
64.5+2.0 66.0+3.5 64.8+2.3 66.1+3.6 65.4+2.9 63.6+1.1 63.5+1.0
24.6−0.1 25.4+0.7 24.9+0.2 24.9+0.2 24.4−0.3 24.3−0.4 23.2−1.5
51.8+3.5 60.6+12.3 59.8+11.5 61.8+13.5 51.9+3.6 52.3+4.0 50.1+1.8
65.7+0.1 66.9+1.3 67.0+1.4 66.2+0.6 67.6+2.0 67.4+1.8 66.5+0.9
85.9 86.1+0.2 84.2−1.7 86.2+0.3 77.4−8.5 88.0+2.1 91.2+5.3
89.3+0.2 89.6+0.5 89.9+0.8 90.2+1.1 88.5−0.6 92.5+3.4 91.6+2.5
70.6−0.1 72.5+1.8 73.3+2.6 72.0+1.3 72.9+2.2 72.7+2.0 71.4+0.7
93.4+0.2 93.4+0.2 92.2−1.0 93.4+0.2 92.4−0.8 94.2+1.0 94.0+0.8
44.0+0.5 45.5+2.0 44.8+1.3 44.9+1.4 45.2+1.7 46.8+3.3 45.3+1.8
67.8+0.3 71.0+3.5 70.3+2.8 69.4+1.9 69.9+2.4 68.8+1.3 68.2+0.7
65.8+0.6 67.7+2.5 67.2+2.0 67.6+2.4 65.9+0.7 67.0+1.7 66.5+1.3
Medium (ξ = 0.01)
TENT TDA DMN OGA ADAPT StatA MOON
66.7+0.1 68.2+1.6 68.0+1.4 68.6+2.0 69.3+2.7 69.6+3.0 76.5+9.9
64.3+1.8 65.6+3.1 64.8+2.3 65.4+2.9 67.2+4.7 65.9+3.4 73.9+11.4
24.6−0.1 25.2+0.5 24.9+0.2 24.9+0.2 24.6−0.1 27.3+2.6 28.3+3.6
47.9−0.4 56.5+8.2 56.2+7.9 58.6+10.3 58.5+10.2 52.3+4.0 51.5+3.2
65.6±0.0 66.5+0.9 66.8+1.2 66.2+0.6 68.6+3.0 73.2+7.6 75.1+9.5
85.9±0.0 85.8−0.1 81.9−4.0 85.9±0.0 83.9−2.0 89.1+3.2 95.3+9.4
89.4+0.3 89.3+0.2 89.0−0.1 89.8+0.7 90.2+1.1 94.6+5.5 94.8+5.7
70.6−0.1 72.6+1.9 73.0+2.3 72.2+1.5 73.2+2.5 75.6+4.9 75.7 +5.0
93.3+0.1 93.5+0.3 92.1−1.1 93.4+0.2 92.5−0.7 94.3+1.1 95.3+2.1
44.0+0.5 45.2+1.7 44.9+1.4 44.8+1.3 45.3+1.8 46.8+3.3 50.0+6.5
67.8+0.3 70.1+2.6 69.6+2.1 69.1+1.6 71.3+3.8 69.7+2.2 73.4+5.9
65.5+0.2 67.1+1.9 66.5+1.2 67.2+1.9 67.7+2.5 68.9+3.7 71.8+6.6
High (ξ = 0.001)
TENT TDA DMN OGA ADAPT StatA MOON
66.8+0.2 67.9+1.3 67.9+1.3 67.9+1.3 68.6+2.0 71.9+5.3 82.2+15.6
64.3+1.8 65.1+2.6 64.8+2.3 64.6+2.1 66.5+4.0 66.0+3.5 77.2+14.7
24.8+0.1 25.1+0.4 24.9+0.2 24.9+0.2 24.6−0.1 27.9+3.2 30.0+5.3
45.6−2.7 55.3+7.0 56.3+8.0 59.0+10.7 55.4+7.1 51.8+3.5 51.6+3.3
65.6±0.0 66.3+0.7 66.8+1.2 66.2+0.6 68.0+2.4 74.7+9.1 77.4+11.8
86.1+0.2 85.5−0.4 79.9−6.0 85.6−0.3 81.3−4.6 89.3+3.4 95.8+9.9
89.4+0.3 89.0−0.1 88.9−0.2 89.7+0.6 89.1±0.0 94.8+5.7 95.3+6.2
70.5−0.2 72.5+1.8 72.9+2.2 72.2+1.5 73.0+2.3 76.4+5.7 77.0+6.3
93.4+0.2 93.6+0.4 92.1−1.1 93.5+0.3 92.5−0.7 94.4+1.2 95.7+2.5
44.0+0.5 45.1+1.6 44.8+1.3 44.7+1.2 45.3+1.8 47.0+3.5 51.2+7.7
67.9+0.4 69.7+2.2 69.4+1.9 68.9+1.4 70.3+2.8 69.8+2.3 74.6+7.1
65.3+0.1 66.8+1.6 66.3+1.0 67.0+1.8 66.8+1.5 69.5+4.2 73.4+8.2
Separate
TENT TDA DMN OGA ADAPT StatA MOON
66.7+0.1 67.4+0.8 67.7+1.1 67.2+0.6 68.2+1.6 71.7+5.1 82.4+15.8
64.2+1.7 64.6+2.1 64.7+2.2 64.1+1.6 65.7+3.2 64.9+2.4 76.7+14.2
24.7±0.0 24.9+0.2 24.9+0.2 24.9+0.2 25.0+0.3 28.9+4.2 32.1+7.4
37.0−11.3 55.3+7.0 55.1+6.8 58.0+9.7 53.7+5.4 48.2−0.1 51.2+2.9
65.6±0.0 65.9+0.3 66.7+1.1 66.1+0.5 67.4+1.8 75.2+9.6 77.8+12.2
86.1+0.2 85.2−0.7 78.5−7.4 85.4−0.5 78.2−7.7 88.9+3.0 95.5+9.6
89.3+0.2 88.9−0.2 88.0−1.1 89.5+0.4 88.2−0.9 95.2+6.1 95.1+6.0
70.8+0.1 72.3+1.6 72.8+2.1 72.1+1.4 72.4+1.7 77.6+6.9 77.9+7.2
93.4+0.2 93.6+0.4 91.9−1.3 93.4+0.2 92.5−0.7 94.3+1.1 95.9+2.7
43.9+0.4 45.0+1.5 44.8+1.3 44.6+1.1 44.8+1.3 45.8+2.3 53.6+10.1
67.9+0.4 69.6+2.1 69.0+1.5 68.7+1.2 69.5+2.0 69.0+1.5 74.6+7.1
64.5−0.7 66.6+1.4 65.8+0.6 66.7+1.5 66.0+0.7 69.1+3.8 73.9+8.6
128 1,000 50,000
8.47 8.99 36.90
0.11 0.31 0.59
0.06 0.13 0.21
αk
Update µ
Update κ
Bs=64
Bs=1,000
✗ ✓ ✓ ✓ ✓
✓ ✗ ✓ ✓ ✓
✓ ✓ ✗ ✓ ✓
✓ ✓ ✓ ✗ ✓
69.0 72.3 66.8 70.8 72.4
68.9 70.4 66.5 68.9 71.1
CF 10
λk
exp(−λk )
81.6 72.9
68.5 67.6
71.1 68.8
update remain noticeably influenced by empirical estimates. These results further support our inverse design αk = 1/λk .
Table 4. Ablation study on components. Each reported performance is averaged over all datasets and scenarios. γ
1/λk (ours)
ImageNet Average
0.03 0.03 0.05
α
TD
ra ft irc
TransCLIP StatA ADAPT MOON 10 10 1 10 0.08 0.24 0.47
Avg.
Table 5. Design choices of αk . Results are reported on batch adaptation, Medium scenario, batch size of 1,000. ”Average” denotes the mean performance over all datasets.
Table 3. Runtime (seconds) on ImageNet. The second row indicates the number of iterations. “CLIP forward” denotes the inference latency, while others report the net algorithm time. Batch size CLIP forward
1
Fl ow
h1 01
Pe ts
er s1 02
Fo od 10 1
or dC a St a
62.5 64.8+2.3
nf
Eu r
oS AT
A
66.6 69.3+2.7
Scenario
39 7 SU N
CLIP MTA
ag e
Method
Im
N et
rs
Table 2. Main results for online adaptation, averaged over 100 runs. The best and second-best results are marked in bold and underlined, respectively. We report four scenarios for batch size 128, with different Dirichlet parameter ξ. Separate denotes sequential classes. Subscript green indicates improvement, red indicates decline, and gray indicates no change compared with zero-shot performance.
We then examine the design of class confidence λk in Tab. 6, using three representative tasks from both extreme and mild imbalance scenarios. Average-only confidence may suppress rare-but-valid classes, while max-only confidence may overestimate classes with occasional high-confidence predictions. Various types of means mitigate this issue similarly. We choose geometric mean as it yields a more generalizable solution across broader scenarios, especially those mild, near-uniform cases where MOON performs below average.
priors for adjustments enables more fine-grained shrinkage in realistic scenarios, with negligible additional cost. Design choices of α and λ. We first analyze the design of the anchor weight αk in Eq. (10). As shown in Tab. 5, directly setting αk = λk performs poorly, since it assigns stronger shrinkage to more confident classes. We also compare an alternative form αk = exp(−λk ), but find it less effective. A possible reason is that it bounds αk in (0, 1], and in turn bounds βk within [nk /(nk + 1), 1), making the
5.4. Further Analysis Comparisons with variants. Although vMF distribution is topologically native to the hypersphere of normalized VLM embeddings, Tab. 7 shows that replacing the GMM in StatA with a vMF mixture unexpectedly underperforms standard StatA. We explain this with the linear representa8
Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction Table 6. Design choices of λk , across extreme and mild imbalance scenarios. Each reported performance is averaged over all datasets. λk
Task 1
Task 2
Task 3
Table 8. Extend MOON to sample-wise. Results are reported on online adaptation, Medium scenario, batch size of 128.
Avg.
Extreme imbalance Geo. mean (ours) Ari. mean Harm. mean Average only Max only
73.1 73.5 73.4 72.4 71.7
72.9 74.2 73.5 73.3 70.8
73.4 74.1 73.8 73.9 71.6
73.1 73.9 73.6 73.2 71.4
Mild imbalance Geo. mean (ours) Ari. mean Harm. mean Average only Max only
69.4 69.0 69.3 68.2 69.0
69.2 68.6 69.0 67.1 68.9
66.5 65.7 66.1 66.7 66.9
68.4 67.8 68.1 67.3 68.3
Batch
StatA StatA vMF MOON Gaussian MOON
Bs=1,000
Bs=128
69.01 68.69
69.46 68.15
68.61 68.31
71.07 71.14
SUN397
Food101
DTD
66.6
62.5
85.9
43.5
StatA MOON MOON†
69.6 76.5 73.0
65.9 73.9 69.5
89.1 95.3 92.4
46.8 50.0 44.1
ImageNet
#Params CLIP RN50 CLIP RN101 CLIP ViT-B/32 CLIP ViT-L/14 OpenCLIP SigLIP EVA-CLIP
Average
Zero-shot
Ours
Zero-shot
Ours
58.2 61.3 62.0 73.5 73.0 82.3 78.0
76.8 +18.6 77.1 +15.8 78.5 +16.5 84.9 +11.4 83.4 +10.4 91.3 +9.0 81.2 +3.2
58.7 59.5 61.9 72.6 72.5 81.8 76.5
65.7 +7.0 64.8 +5.3 68.0 +6.1 77.4 +4.8 76.7 +4.2 85.1 +3.2 78.4 +1.9
102M 120M 151M 428M 150M 878M 1.14B
Online
Bs=64
71.49 72.36
ImageNet
CLIP
Table 9. Scaling to other CLIP backbones and VLMs.
Table 7. Comparisons with variants. Each reported performance is averaged over all datasets and scenarios. Method
Method
available data in B. Therefore, the PLE objective becomes5 : B min −z⊤ ⋆ log p⋆ −
z⋆ ∈∆K
71.23 71.42
X
ω⋆j , z⊤ ⋆ z̄j + KL(z⋆ ∥ŷ⋆ ).
(16)
j∈B
Solving the problem yields the closed-form predictor: P ŷ⋆ ⊙ exp(log pB ω⋆j z̄j ) ⋆ + P j∈B . z⋆ = ⊤ ω (ŷ⋆ ⊙ exp(log pB + ⋆ j∈B ⋆j z̄j )) 1K
tion hypothesis (Park et al., 2023), which suggests that VLM representations reside in low-dimensional linear subspaces and exhibit strong anisotropy. Consequently, constrained by isotropic scalar κ, vMF struggles to capture such manifold geometry, while Gaussian covariance implicitly approximates it in Euclidean space. However, in MOON, this limitation is largely alleviated with our dynamic shrinkage strength modeling: MOON achieves comparable performance to its Gaussian variant with much higher efficiency.
(17)
Hence, MOON can make an immediate prediction for a newly arriving sample without rerunning full-batch transduction. At the same time, using bank statistics, the mixture parameters continue to be stabilized by the KL-anchored dynamic shrinkage mechanism. Specifically, the parameter update keeps the same anchored form as Eq. (14): µB k =
Extend MOON to sample-wise. While online TTA methods usually assume access to only a single image at each step4 , vanilla MOON does not support sample-wise mode, as transductive learning inherently requires a batch of samples to perform probabilistic soft clustering. However, MOON can be easily extended to a sample-wise online mode by introducing a memory bank to collect historical samples, as what recent work (Zhang et al., 2025) has done.
B ′ βkB v B k + (1 − βk )µk , B B B ∥βk v k + (1 − βk )µ′k ∥
(18)
B B B ′ Ad (κB k ) = ∥βk v k + (1 − βk )µk ∥,
P
γj,k z̄j,k fj
P
γj,k z̄j,k
Pj∈B where v B and βkB = P j∈B . k = j∈B γj,k z̄j,k j∈B γj,k z̄j,k +αk According to this, we establish a sample-wise version MOON† in Tab. 8. As shown, MOON† could still achieve excellent performance. In practice, cold-starting the memory bank B with a held-out set may lead to better results.
Formally, assume a memory bank B = (fj , z̄j ), where fj ∈ S d−1 is cached historical feature, and z̄j ∈ ∆K is the corresponding soft label. Consider a newly arriving sample with normalized feature f⋆ and zero-shot prediction ŷ⋆ . Assume that current mixture parameters B K MB = {(µB k , κk )}k=1 have already been estimated from
More backbones and architectures. We further extend our evaluation to include 4 additional CLIP backbones and 3 other VLMs in Tab. 9, which demonstrate the universal effectiveness of our MOON across diverse model architectures, scales, and types. App. I.2 and I.3 provide more details.
4 For consistency, we assume a batch-wise access of test samples for all methods in this paper.
5 Note that the KL anchor term R(M) now becomes constant with respect to new assignment z⋆ .
9
Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction
6. Conclusion
Cao, Z., Ma, L., Long, M., and Wang, J. Partial adversarial domain adaptation. In Proceedings of the European conference on computer vision (ECCV), pp. 135–150, 2018.
In this work, we systematically revisit test-time transduction for VLMs under realistic class imbalance from the perspective of PLE, and reveal the brittleness and underlying limitations of existing transductive methods. Therefore, we propose MOON, which is based on a mixture of vMF distributions and dynamically adjusts shrinkage strength at both the instance and class levels to mitigate negative transfer. Extensive experiments validate that MOON achieves effective, efficient and practical adaptation.
Chen, X., Huang, J., Liu, Z., Jiang, Q., Huang, F., Jiang, J., and Wang, Z. Test-time distillation for continual model adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7593– 7604, 2026. Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., and Jitsev, J. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2818–2829, 2023.
Acknowledgment Ziyue Qiao is supported by the National Natural Science Foundation of China (No. 62406056) and the Guangdong Basic and Applied Basic Research Foundation (No. 2024A1515140114). The authors are grateful to the anonymous reviewers for their efforts and insightful suggestions to improve this paper.
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606–3613, 2014.
Impact Statement
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255. Ieee, 2009.
This work advances test-time adaptation for vision-language models by addressing realistic class imbalance and distribution shifts. Our approach leverages a mixture of von MisesFisher distributions with dynamic, data-driven shrinkage to suppress unreliable predictions and mitigate negative transfer. We provide theoretical grounding through penalized likelihood estimation, and demonstrate empirical effectiveness and efficiency across multiple datasets and backbones. By enabling robust and efficient adaptation, our method improves the reliability of deployed vision-language systems in practical applications such as image classification, retrieval, and zero-shot reasoning, without requiring access to model weights or additional training data.
Döbler, M., Marsden, R. A., Raichle, T., and Yang, B. A lost opportunity for vision-language models: a comparative study of online test-time adaptation for vision-language models. In European Conference on Computer Vision, pp. 117–133. Springer, 2024. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
References Abdul Samadh, J., Gani, M. H., Hussein, N., Khattak, M. U., Naseer, M. M., Shahbaz Khan, F., and Khan, S. H. Align your prompts: Test-time prompting with distribution alignment for zero-shot generalization. Advances in Neural Information Processing Systems, 36: 80396–80413, 2023.
Fei-Fei, L., Fergus, R., and Perona, P. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pp. 178–178. IEEE, 2004.
Banerjee, A., Dhillon, I. S., Ghosh, J., Sra, S., and Ridgeway, G. Clustering on the unit hypersphere using von mises-fisher distributions. Journal of Machine Learning Research, 6(9), 2005.
Feng, C.-M., Yu, K., Liu, Y., Khan, S., and Zuo, W. Diverse data augmentation with diffusions for effective test-time prompt tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2704–2714, 2023.
Bossard, L., Guillaumin, M., and Van Gool, L. Food-101– mining discriminative components with random forests. In European conference on computer vision, pp. 446–461. Springer, 2014.
Fuchs, C., Zanella, M., and De Vleeschouwer, C. Online gaussian test-time adaptation of vision-language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 128–137, 2025. 10
Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction
Gong, T., Jeong, J., Kim, T., Kim, Y., Shin, J., and Lee, S.J. NOTE: Robust continual test-time adaptation against temporal correlation. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Li, Q., Liu, Z., Luo, W., Luo, T., and Hou, C. Correcting visual blur induced by attention distraction to reduce hallucinations: Algorithm and theory. In Forty-third International Conference on Machine Learning, 2026a.
Gopal, S. and Yang, Y. Von mises-fisher clustering models. In International conference on machine learning, pp. 154– 162. PMLR, 2014.
Li, Q., Liu, Z., Xu, T., Luo, T., and Hou, C. Adaptive disentangled representation learning for incomplete multi-view multi-label classification. arXiv preprint arXiv:2601.05785, 2026b.
Govindarajan, H., Sidén, P., Roll, J., and Lindsten, F. Dino as a von mises-fisher mixture model. arXiv preprint arXiv:2405.10939, 2024.
Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916, 2023.
Han, Z., Yang, J., Wang, G., Li, J., Xu, Q., Shou, M. Z., and Zhang, C. Dota: Distributional test-time adaptation of vision-language models. arXiv preprint arXiv:2409.19375, 2024.
Liu, J., Song, L., and Qin, Y. Prototype rectification for few-shot learning. In European conference on computer vision, pp. 741–756. Springer, 2020.
Hasnat, M. A., Bohné, J., Milgram, J., Gentric, S., and Chen, L. von mises-fisher mixture model-based deep learning: Application to face verification. arXiv preprint arXiv:1706.04264, 2017.
Liu, Z., Wei, Y., Feng, L., Su, X., Xia, X., Guan, W., Xie, Z., and Yang, S. Do all individual layers help? an empirical study of task-interfering layers in vision-language models. arXiv preprint arXiv:2602.01167, 2026.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.
Lozier, D. W. Nist digital library of mathematical functions. Annals of Mathematics and Artificial Intelligence, 38(1): 105–119, 2003. Lu, Y., Liu, J., Zhang, Y., Liu, Y., and Tian, X. Prompt distribution learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5206–5215, 2022.
Helber, P., Bischke, B., Dengel, A., and Borth, D. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217–2226, 2019.
Maji, S., Rahtu, E., Kannala, J., Blaschko, M., and Vedaldi, A. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151, 2013.
Huang, J., Chen, X., Liu, Z., Sun, Y., Jiang, J., and Wang, Z. What drives test-time adaptation for clip? a controlled empirical study from an update perspective. arXiv preprint arXiv:2606.14299, 2026. Ilharco, G., Wortsman, M., Carlini, N., Taori, R., Dave, A., Shankar, V., Namkoong, H., Miller, J., Hajishirzi, H., Farhadi, A., et al. Openclip. Zenodo, 2021.
Martin, S., Boudiaf, M., Chouzenoux, E., Pesquet, J.-C., and Ayed, I. Towards practical few-shot query sets: Transductive minimum description length inference. Advances in Neural Information Processing Systems, 35:34677– 34688, 2022.
Kalantidis, Y., Tolias, G., et al. Label propagation for zeroshot classification with vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23209–23218, 2024.
Martin, S., Huang, Y., Shakeri, F., Pesquet, J.-C., and Ben Ayed, I. Transductive zero-shot and few-shot clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 28816–28826, 2024.
Karmanov, A., Guan, D., Lu, S., El Saddik, A., and Xing, E. Efficient test-time adaptation of vision-language models. The IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Nilsback, M.-E. and Zisserman, A. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pp. 722–729. IEEE, 2008.
Krause, J., Stark, M., Deng, J., and Fei-Fei, L. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pp. 554–561, 2013.
Ochal, M., Patacchiola, M., Vazquez, J., Storkey, A., and Wang, S. Few-shot learning with class imbalance. IEEE Transactions on Artificial Intelligence, 4(5):1348–1358, 2023. 11
Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction
Park, K., Choe, Y. J., and Veitch, V. The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311.03658, 2023.
Yuan, L., Xie, B., and Li, S. Robust test-time adaptation in dynamic scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15922–15932, 2023.
Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3498–3505. IEEE, 2012.
Zanella, M. and Ben Ayed, I. On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23783–23793, 2024.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748–8763. PmLR, 2021.
Zanella, M., Gérin, B., and Ayed, I. Boosting visionlanguage models with transduction. Advances in Neural Information Processing Systems, 37:62223–62256, 2024.
Razaviyayn, M., Hong, M., and Luo, Z.-Q. A unified convergence analysis of block successive minimization methods for nonsmooth optimization. SIAM Journal on Optimization, 23(2):1126–1153, 2013.
Zanella, M., Fuchs, C., De Vleeschouwer, C., and Ben Ayed, I. Realistic test-time adaptation of vision-language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 25103–25112, 2025.
Shu, M., Nie, W., Huang, D.-A., Yu, Z., Goldstein, T., Anandkumar, A., and Xiao, C. Test-time prompt tuning for zero-shot generalization in vision-language models. Advances in Neural Information Processing Systems, 35: 14274–14289, 2022.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 11975–11986, 2023. Zhang, R., Zhang, W., Fang, R., Gao, P., Li, K., Dai, J., Qiao, Y., and Li, H. Tip-adapter: Training-free adaption of clip for few-shot classification. In European conference on computer vision, pp. 493–510. Springer, 2022.
Soomro, K., Zamir, A. R., and Shah, M. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012.
Zhang, T., Wang, J., Guo, H., Dai, T., Chen, B., and Xia, S.-T. Boostadapter: Improving test-time adaptation via regional bootstrapping. arXiv preprint arXiv:2410.15430, 2024a.
Sun, Q., Fang, Y., Wu, L., Wang, X., and Cao, Y. Evaclip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389, 2023. Veilleux, O., Boudiaf, M., Piantanida, P., and Ben Ayed, I. Realistic evaluation of transductive few-shot learning. Advances in Neural Information Processing Systems, 34: 9290–9302, 2021.
Zhang, Y., Zhu, W., Tang, H., Ma, Z., Zhou, K., and Zhang, L. Dual memory networks: A versatile adaptation approach for vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 28718–28728, 2024b.
Wang, D., Shelhamer, E., Liu, S., Olshausen, B., and Darrell, T. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006.10726, 2020.
Zhang, Y., Kim, Y., Choi, Y.-G., Kim, H., Liu, H., and Hong, S. Backpropagation-free test-time adaptation via probabilistic gaussian alignment. arXiv preprint arXiv:2508.15568, 2025.
Wang, Z., Dai, Z., Póczos, B., and Carbonell, J. Characterizing and avoiding negative transfer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11293–11302, 2019.
Zhao, H., Liu, Y., Alahi, A., and Lin, T. On pitfalls of test-time adaptation. arXiv preprint arXiv:2306.03536, 2023.
Wang, Z., Liang, J., Sheng, L., He, R., Wang, Z., and Tan, T. A hard-to-beat baseline for training-free clip-based adaptation. arXiv preprint arXiv:2402.04087, 2024.
Zou, H. The adaptive lasso and its oracle properties. Journal of the American statistical association, 101(476):1418– 1429, 2006.
Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralba, A. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 3485–3492. IEEE, 2010. 12
Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction
A. Overall Procedure of MOON The overall procedure of our proposed MOON is presented in Alg. 1. The BSUM-style iterative optimization of MOON can be conceptualized as a generalized Expectation-Maximization (EM) algorithm: fixing parameters (µk , κk ) to update assignments z corresponds to the E-step, while fixing z and updating (µk , κk ) corresponds to the M-step. Algorithm 1 Overall procedure of MOON d−1 d−1 Require: Visual feature embeddings {fi }N , textual class embeddings {tk }K ; Fixed hyperpai=1 , fi ∈ S k=1 , tk ∈ S rameters (no need for tuning): VLM temperature τ , Number of neighbors in Laplacian term m, Iterations T . Ensure: Latent label assignments as final predictions z ∈ [0, 1]N ×K . # Initialization exp(fi⊤ tk /τ ) P ; initialize assignments z ← ŷ. ▷ See Eq. (1) 1: Compute zero-shot logits ŷ = {ŷi,k }K k=1 , ŷi,k = exp(f ⊤ tj /τ ) j
i
K 2: Initialize vMF parameters µ = {µk }K µ′ and κ′ . k=1 and κ = {κk }k=1 ; prior anchors (
▷ See Eq. (7) if fj is the m-nearest neighbors of fi 3: Build a m-NN affinity graph W = [ωi,j ] ∈ RN ×N , where ωi,j = . 0, otherwise K 4: Compute class-level adjustment weights α = {αk }K ▷ See Eq. (10) and (11) k=1 with confidence λ = {λk }k=1 . N 5: Initialize instance-level adjustment weights γ = {γi }i=1 . ▷ See Eq. (9) # BSUM-style iterative optimization 6: for t = 1 to T do # Block update with respect to assignments z ⊤ 7: Compute vMF log-likelihood scores: log pvMF ▷ See Eq. (6) i,k ← κk µk fi + log Cd (κk ). f⊤ i fj ,
8: 9: 10:
11:
P (t) ŷi ⊙exp(log pvMF + j ωij zj ) i . P (t) ⊤ vMF (ŷi ⊙exp(log pi + j ωij zj )) 1K H(zi ) Update instance-level adjustment weights: γi = 1 − log K . (t+1)
Update z by single-pass linear approximation: zi
=
▷ See Eq. (12)
# Compute shrinkage strengths PN k Compute shrinkage strengths β = {βk }K βk ← nkn+α . i,k , k=1 : nk ← i=1 γi zP k (In practice, we use hard label counts for stability: nk = i γi I[arg maxj zi,j = k].) # Block update with respect to distribution parameters (µ,Pκ) γ z fi . Compute empirical estimates from standard MLE: vk ← Pi γi i zi,k i,k i
′
k )µk , Ad (κk ) = ∥βk v k + (1 − βk )µ′k ∥, ▷ Update µ and Ad (κ) by closed-form anchor shrinkage: µk = ∥ββkk vvkk +(1−β +(1−βk )µ′k ∥ See Eq. (14) 13: Inversely estimate κ using Banerjee approximation: κk = A−1 ▷ See Eq. (15) d (κk ). 14: end for 15: return z.
12:
In implementation, we modify the shrinkage strength βk by replacing the soft assignment predictions with hard ones on the vertices of the probability simplex ∆K following StatA (Zanella et al., 2025), which is:
1 [k = argmaxr γi,k zi,r ] . βk ≈ PN i=1 i=1 1 [k = argmaxr γi,k zi,r ] + αk PN
(19)
This design has been proven to be more robust in practice, with experimental results shown in App. J. Complexity analysis. We provide a theoretical complexity analysis of our vMF-based MOON and Gaussian-based stateof-the-art method StatA, further demonstrating the significant efficiency advantage of vMF mixtures that has been largely overlooked in prior works. Formally, let N, K, d denote the number of samples, classes, and feature dimensions, respectively. Let m be the number of neighbors and T the number of iterations. (1) Graph construction: Both methods share the initial cost of constructing the affinity graph. This requires O(N 2 d) for dense similarity computation, which dominates the pre-processing cost. (2) Per-iteration cost: For MOON, each iteration is dominated by two parts: (i) likelihood computation and parameter aggregation, which scales with O(N Kd); and (ii) sparse graph smoothing, which scales with O(N mK). Thus, the 13
Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction
per-iteration complexity is O(N K(d + m)). In contrast, StatA incurs higher costs due to two factors: (i) it performs LG inner loops for assignment updates (typically LG = 5), inflating the smoothing cost to O(LG N mK); and (ii) its parameter update involves computing per-class covariance matrices, adding an O(Kd2 ) term. (3) Total cost: Excluding the shared graph construction, the total asymptotic complexities are: OMOON ≈ T · N K(d + m)
OStatA ≈ T · (N Kd + LG N mK + Kd2 ).
vs.
(20)
MOON achieves superior efficiency by avoiding the quadratic complexity O(d2 ) in parameter updates and eliminating the need for inner assignment loops. Convergence guarantee of optimization algorithm. Our algorithm can be analyzed within the standard BSUM framework. Let L(z, Θ) denote the objective and Θ = {µk , κk }K k=1 . Given that: (1) Affinity matrix W is PSD; (2) Anchor weights αk and γi > 0; (3) Fixing z, the parameter subproblem in Θ has a unique minimizer. Then each outer iteration of our algorithm is a valid BSUM step: • z-update: The only non-convex part is the Laplacian term. Since W is PSD, this term is concave and can be upperbounded by its first-order Taylor expansion at the current iteration (proofs in App. G.1). Minimizing this tight surrogate yields Eq. (12), i.e., the update is an exact minimizer of the majorized subproblem. • Θ update: Fixing z, our objective is strictly convex, and Eqs. (13)-(14) are the corresponding closed-form exact updates. This step further decreases the objective. Therefore, each outer iteration satisfies L(z (t+1) , Θ(t+1) ) ≤ L(z (t) , Θ(t) ).
(21)
Since La is lower-bounded, the objective sequence is monotonically non-increasing and thus convergent. And, every limit point of the iterate sequence is a coordinate-wise minimum. And, as BSUM doe not require solving subproblems to full convergence at each outer iteration, single-pass z-update preserves convergence guarantee.
B. Revisiting StatA StatA (Zanella et al., 2025) first discuss transduction under the realistic class imbalanced scenarios at test-time. Unlike our MOON which operates on the unit hypersphere, StatA assumes that the visual feature representations {fi }N i=1 follow a Gaussian Mixture Model (GMM) in the Euclidean space. To handle performance degradation, it introduces a KL-divergencebased regularization term acting as a statistical anchor. The optimization objective is to minimize the PLE objective penalized by the deviation from zero-shot prior anchors leveraged from text {N (µ′k , Σ′k )}K k=1 : LStatA (z, θ) = −
N X
z⊤ i log N (fi |µk , Σk ) +R(z) + α
i=1
| where
R(z) = −
}
negative log-likelihood (NLL)
ωij z⊤ i zj +
i,j
|
KL N (µ′k , Σ′k )∥N (µk , Σk ) ,
k=1
{z
X
K X
N X
|
{z
statistical anchor
} (22)
KL(zi ∥ŷi ) .
i=1
{z
Laplacian reg.
}
|
{z
text supervision
}
where θ = {µk , Σk }K and α > 0 is a hyperparameter as anchor term weight. The anchor k=1 are the mixture parameters, distribution is initialized as: µ′k = tk and Σ′ = Diag
P
(fi −µk )(fi −µk ) i,k ŷi,kP i,k ŷi,k
⊤
.
The algorithm is performed via BSUM-style iterative optimization, which alternates between assignments z block and distribution parameters θ block: 14
Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction
• Assignment update: Fixing θ, the assignments zi,k are updated based on the Gaussian likelihoods pi,k ∝ √ 1 exp − 12 (fi − µk )⊤ Σ−1 k (fi − µk ) , similar rule as Eq. (12). |Σk |
• Parameter update: Fixing z, the parameters are updated in closed form. Crucially, the update for the mean µk and covariance Σk exhibits an anchor shrinkage behavior: µk = βk vk + (1 − βk )µ′k ,
Σk = βk Tk + (1 − βk )(Σ′ + Diag((µ′k − µk )2 )),
(23)
with empirical estimates as: PN zi,k fi ; v k = Pi=1 N i=1 zi,k Here, nk = strength.
PN Tk =
2 i=1 zi,k Diag((fi − µk ) ) . PN i=1 zi,k P
1[k=argmax z
(24)
]
nk r i,r P i i zi,k is the soft count, and βk = nk +α ≈ 1[k=argmax zi,r ]+α ∈ [0, 1] denotes the shrinkage
P
i
r
Limitations. As shown in Eq. (23), the shrinkage strength βk , controlled by soft count nk and anchor weight α, balances the trade-off between empirical estimate vk and the prior anchor µ′k . However, anchor weight α is a fixed scalar hyperparameter shared across all classes and samples. This implies a static shrinkage strength modeling that ignores the varying reliability of different classes (e.g., outliers vs. effective classes) or instances, rendering StatA suboptimal in both accuracy and robustness under realistic class imbalance, as discussed in Sec. 1.
C. Experimental Details C.1. Datasets We evaluate our proposed MOON and other baselines on 11 widely-used public datasets for fine-grained visual classification. These datasets cover a diverse range of domains, including generic objects, scenes, textures, satellite imagery, and specific fine-grained categories. Specifically, the benchmark includes: ImageNet (Deng et al., 2009), SUN397 (Xiao et al., 2010), Aircraft (Maji et al., 2013), EuroSAT (Helber et al., 2019), StanfordCars (Krause et al., 2013), Food101 (Bossard et al., 2014), Pets (Parkhi et al., 2012), Flowers102 (Nilsback & Zisserman, 2008), Caltech101 (Fei-Fei et al., 2004), DTD (Cimpoi et al., 2014), and UCF101 (Soomro et al., 2012). Detailed statistics for these datasets are provided in Tab. 10. C.2. Baselines We compare our MOON against a comprehensive set of baselines, which are categorized into: (1) Transductive methods: EMDirichlet (Dirichlet) (Martin et al., 2024), ZLaP (Kalantidis et al., 2024), GDA-CLIP (Wang et al., 2024), TransCLIP (Zanella et al., 2024), ADAPT (Zhang et al., 2025), and StatA (Zanella et al., 2025)6 ; (2) Online TTA methods: TENT (Wang et al., 2020), TDA (Karmanov et al., 2024), DMN (Zhang et al., 2024b), and OGA (Fuchs et al., 2025). We also incorporate another TTA method MTA (Zanella & Ben Ayed, 2024) that requires per-image augmentation. The details and experimental configurations of these baselines are listed as below: • ZLaP (CVPR’24): introduces a non-parametric framework that leverages the graph structure of unlabeled data via label propagation, utilizing geodesic distances on the data manifold to address the modality gap in VLMs. The number of nearest neighbors m is set to 5, scale parameter of RBF kernel function γ is set to 5.0, and the clamping factor α is fixed at 0.3. We don’t specifically scale the similarity matrix. • EM-Dirichlet (CVPR’24): frames transduction on the unit simplex by modeling class-conditional feature distributions with a Dirichlet law, solving the MLE problem via a hyperparameter-free Block Majorization-Minimization algorithm. The temperature T in the probabilities is fixed to 30. • GDA-CLIP (ICLR’24): applies Gaussian Discriminant Analysis (GDA) by assuming a shared covariance matrix for class features, estimating parameters via closed-form solutions and ensembling them with zero-shot logits. The ensemble weight is set to α by default. 6
Since StatA and ADAPT are also evaluated under online settings, we provide their corresponding results in the main manuscript.
15
Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction
• TransCLIP (NeurIPS’24): formulates adaptation as a regularized MLE with a text-guided KL-divergence penalty, employing an iterative optimization procedure that decouples sample assignments and parameter updates. Here, text-guided KL divergence penalty λ is set to 1, and the number of nearest neighbors m is set to 3. • ADAPT (NeurIPS’25): presents a backpropagation-free method that reframes adaptation as Gaussian probabilistic inference with closed-form updates, utilizing a knowledge bank to efficiently support online and transductive settings. We set the bank size L to 12, and the momentum coefficient for parameter update α to 0.9. • StatA (CVPR’25): addresses realistic scenarios with variable effective classes by employing a statistical anchor regularization within a Gaussian Mixture Model (GMM) to dynamically constrain features near text-derived priors. The anchor term weight α is set to 1, with the number of nearest neighbors m set to 3. We use hard βk by default. • TENT (ICLR’21): adapts the model by minimizing the Shannon entropy of predictions on target data, specifically updating the affine parameters of Batch Normalization layers to align internal statistics online. We set the learning rate to 1e-3, and perform 5 adaptation steps for each batch. • TDA (CVPR’24): utilizes a lightweight key-value cache system with entropy-based filtering and introduces a negative cache mechanism to explicitly penalize unlikely classes using negative pseudo-labeling. All the configuration is kept the same as those set in the original paper on ImageNet. • DMN (CVPR’24): integrates a static memory for pre-trained knowledge and a dynamic memory for historical test features, employing a cross-attention strategy to refine decision boundaries based on temporal context. All the configuration is kept the same as those set in the original paper on ImageNet. • OGA (CVPRW’25): reframes online adaptation as a Maximum A Posteriori (MAP) estimation problem using multivariate Gaussian distributions and zero-shot priors to calibrate predictions without gradient backpropagation. THe memory update threshold τ is set to 0.01, with cache memory capacity set to 8. • MTA (CVPR’24): proposes a training-free strategy that leverages MeanShift on augmented views to identify distribution modes, jointly optimizing a learnable inlierness score to robustly aggregate visual information. All the configuration is kept the same as those set in the original paper on ImageNet. C.3. Prompts Following (Zhang et al., 2022), we adopt default, fixed prompt templates to initialize text embeddings for all methods, as illustrated in Tab. 10. Table 10. Dataset information and prompt templates. Name
Other name
#K
#N
Description
Prompt template
SUN397 Aircraft EuroSAT StanfordCars Food101 Pets Flowers102 Caltech101 DTD UCF101 ImageNet
SUN397 FGVCAircraft EuroSAT Cars Food101 OxfordPets OxfordFlowers Caltech101 DTD UCF101 ImageNet-1K
397 100 10 196 101 37 102 101 47 101 1000
19,850 3,333 8,100 8,041 30,300 3,669 2,463 2,465 1,692 3,783 50,000
Scenes classification Aircraft classification Satellite images classification Cars classification Food classification Pets classification Flowers classification Objects classification Textures classification Actions classification Objects classification
"a photo of a [ ]." "a photo of a [ ], a type of aircraft." "a centered satellite photo of [ ]." "a photo of a [ ]." "a photo of [ ], a type of food." "a photo of [ ], a type of pet." "a photo of a [ ], a type of flower." "a photo of a [ ]." "[ ] texture." "a photo of a person doing [ ]." "a photo of a [ ]."
C.4. Data Sampler In this section, we describe the sampling strategies for constructing realistic test-time scenarios, following the protocols in StatA (Zanella et al., 2025). 16
Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction
Batch adaptation. To simulate realistic class sparsity where the label distribution within a batch is partial, we construct test batches with a limited number of effective classes. Specifically, given a batch size B and a total of K classes, we first min max determine the number of effective classes Kef f , which is either fixed or uniformly sampled from [Kef f , Kef f ]. We then randomly select a subset of classes Cbatch with size Kef f and aggregate all their corresponding samples. The final test batch is formed by randomly sampling B instances without replacement from this restricted pool, ensuring that the batch contains only a fraction of the total categories. Online adaptation. We generate non-i.i.d. data streams using a Dirichlet-based framework to evaluate robustness against temporal correlation (Yuan et al., 2023). The data stream is divided into slots, where the allocation of each class across slots follows a Dirichlet distribution Dir(ξ · 1). The scalar ξ controls the correlation intensity: large values approximate an i.i.d. stream, while small values concentrate classes into fewer slots to create high temporal correlation. Additionally, we consider a separate sequential setting (simulating ξ → 0), where classes are randomly permuted and all samples from a class appear contiguously before transitioning to the next, representing the most extreme temporal correlation. C.5. Implementation Details Unless otherwise specified, we employ CLIP ViT-B/16 as the default backbone and evaluate performance using the Top-1 accuracy. Consistent with our black-box assumption, we utilize a fixed set of hyperparameters across all experiments without per-task tuning: we set the nearest neighbors m = 3 and the number of iterations to 10, while inheriting the temperature parameter τ directly from pre-trained VLM. For stability and robustness, we employ hard assignments for the shrinkage strength βk , and dynamically update the instance-level weights γi at each iteration. All experiments are conducted on a single NVIDIA RTX 4090 24GB GPU. To ensure statistical reliability given the stochastic data sampling, all reported results represent the average of 1,000 independent runs for batch adaptation and 100 runs for online adaptation, initialized with a fixed random seed of 1.
D. Generality of KL-Anchored PLE for Exponential Families In this section, we provide a formal proof that KL-based distribution anchor in the penalized likelihood estimation (PLE) formulation, i.e., R(M) in Eq. (2), yields an adaptive shrinkage behavior that enables convex combination update in the mean-parameter space for any (regular) exponential-family class-conditional mixture model. D.1. Problem Setup Consider a K-class latent-variable mixture model with unlabeled samples {xi }N i=1 and soft assignments zi,k ∈ [0, 1] PK satisfying k=1 zi,k = 1. For each class k ∈ {1, . . . , K}, assume the class-conditional density function belongs to a regular minimal exponential family: p(x | η) = h(x) exp η ⊤ T (x) − A(η) , (25) where η is the natural parameter, T (x) is the sufficient statistic, and A(η) is the log-partition function. For a regular minimal exponential family, A(η) is strictly convex and the mapping ∇A is one-to-one, relating natural parameters to mean parameters µ = Eη [T (X)] = ∇A(η). Given soft assignments, define the class-wise soft counts and soft sufficient-statistic sums:
nk =
N X
zi,k ,
Sk =
i=1
N X
zi,k T (xi ).
(26)
i=1
Let qk (x) = p(x | ηk′ ) denote a fixed anchor distribution for class k. Conditioned on {zi,k }, the M-step minimizes the following KL-anchored PLE objective over {ηk }K k=1 :
L({ηk }) = −
N X K X
zi,k log p(xi | ηk ) + α
i=1 k=1
K X k=1
17
KL qk ∥ p(· | ηk ) ,
α > 0.
(27)
Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction
D.2. Closed-Form KL for Exponential Families Lemma D.1 (KL divergence within an exponential family). Let q(·) = p(· | η ′ ) and p(·) = p(· | η) be members of the same exponential family (25). Then KL p(· | η ′ ) ∥ p(· | η) = A(η) − A(η ′ ) − (η − η ′ )⊤ µ′ , µ′ = Eη′ [T (X)] = ∇A(η ′ ). (28) Proof. By definition, KL p(· | η ′ ) ∥ p(· | η) = Eη′ log p(X | η ′ ) − log p(X | η) .
(29)
log p(X | η ′ ) − log p(X | η) = (η ′ )⊤ T (X) − A(η ′ ) − η ⊤ T (X) − A(η) ,
(30)
Using (25), we have
since log h(X) cancels. Taking expectation under p(· | η ′ ) yields KL p(· | η ′ ) ∥ p(· | η) = (η ′ − η)⊤ Eη′ [T (X)] + A(η) − A(η ′ ) = A(η) − A(η ′ ) − (η − η ′ )⊤ µ′ ,
(31) (32)
where µ′ = Eη′ [T (X)] = ∇A(η ′ ). D.3. KL Anchoring Implies Convex Combination in Mean-Parameter Space Theorem D.2 (KL-anchored M-step yields convex combination in mean space). Assume each class-conditional model p(· | ηk ) belongs to a regular minimal exponential family (25). Fixing soft assignments {zi,k }, the M-step objective (27) is strictly convex in each ηk and admits a unique minimizer ηk⋆ . Moreover, the corresponding mean parameter satisfies ∇A(ηk⋆ ) =
Sk + αµ′k , nk + α
µ′k = ∇A(ηk′ ).
(33)
Equivalently, for nk > 0 with empirical mean parameter µ̂k = Sk /nk , ∇A(ηk⋆ ) = βk µ̂k + (1 − βk )µ′k , Proof. The objective (27) decomposes across classes: L({ηk }) = log h(x) + η ⊤ T (x) − A(η), −
N X i=1
zi,k log p(xi | ηk ) = −
N X
βk =
nk ∈ [0, 1]. nk + α
(34)
PK
k=1 Lk (ηk ) + const, where, using log p(x | η) =
zi,k log h(xi ) + ηk⊤ T (xi ) − A(ηk )
(35)
i=1
= −ηk⊤
N X
zi,k T (xi ) +
i=1
N X
zi,k A(ηk ) + const
(36)
i=1
= −Sk⊤ ηk + nk A(ηk ) + const. For the KL anchor term, apply Lemma D.1 with qk (·) = p(· | ηk′ ) and p(· | ηk ): KL qk ∥ p(· | ηk ) = A(ηk ) − A(ηk′ ) − (ηk − ηk′ )⊤ µ′k ,
(37)
µ′k = ∇A(ηk′ ).
Combining (37) and (38) (and dropping constants independent of ηk ), we obtain Lk (ηk ) = −Sk⊤ ηk + nk A(ηk ) + α A(ηk ) − (ηk )⊤ µ′k + const = −(Sk + αµ′k )⊤ ηk + (nk + α)A(ηk ) + const. 18
(38)
(39) (40)
Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction
Since A(η) is strictly convex for a regular minimal exponential family and nk + α > 0, it follows that Lk (ηk ) is strictly convex in ηk and thus has a unique minimizer ηk⋆ . Taking the gradient of (40) and setting it to zero yields the first-order optimality condition: −(Sk + αµ′k ) + (nk + α)∇A(ηk⋆ ) = 0,
(41)
which implies (33). When nk > 0, substituting Sk = nk µ̂k into (33) gives (34) with βk = nk /(nk + α) ∈ [0, 1]. D.4. Corollaries for Realistic Test-Time Scenarios Theorem D.2 immediately yields two properties that are particularly relevant to realistic test-time settings with class imbalance and sparse effective label coverage. Corollary D.3 (No deviation for outlier classes). If a class is entirely outlier (absent classes) in the current batch in the sense that nk = 0, then the optimal mean parameter satisfies ∇A(ηk⋆ ) = µ′k = ∇A(ηk′ ).
(42)
Moreover, since ∇A is injective for a regular minimal exponential family, it follows that ηk⋆ = ηk′ .
(43)
That is, the optimal parameter for an outlier (absent) class is exactly its anchor parameter, and will not deviate due to the absence of evidence. Proof. Setting nk = 0 and Sk = 0 in (33) gives ∇A(ηk⋆ ) = µ′k . Injectivity of ∇A in a regular minimal exponential family implies ηk⋆ = ηk′ . Corollary D.4 (Bounded deviation for rare classes). For any class with nk > 0, the deviation from the anchor in meanparameter space is bounded by ∇A(ηk⋆ ) − µ′k = βk µ̂k − µ′k ≤
nk µ̂k − µ′k . nk + α
(44)
Thus, when a class appears rarely (small nk ), its update magnitude away from the anchor is linearly shrunk by βk . Proof. Equation (34) implies ∇A(ηk⋆ ) − µ′k = βk (µ̂k − µ′k ). Taking norms on both sides yields (44). Remarks. Theorem D.2 formalizes a key mechanism behind KL-anchored PLE: regardless of the specific exponentialfamily choice, the anchor regularization transforms the M-step into a strictly convex problem whose first-order condition yields an adaptive shrinkage that enables convex combination in mean-parameter space. This property is particularly beneficial in realistic test-time adaptation, where many classes may be missing or under-represented within a batch (e.g., Keff ≪ K and ξ → 0), and naive maximum-likelihood estimation tends to overfit to locally biased statistics.
E. Details of von Mises-Fisher Distributions E.1. Introduction to vMF distributions Let x ∈ Rd be a random vector on the unit hypersphere Sd−1 , i.e., ∥x∥2 = 1. The probability density function of von Mises-Fisher (vMF) distribution on Sd−1 is defined by p(x; µ, κ) = Cd (κ) exp κ µ⊤ x , µ ∈ Sd−1 , κ ≥ 0, (45) where µ is the mean direction vector and κ is the concentration scalar. The normalization constant is given by Cd (κ) =
κν (2π)d/2 Iν (κ) 19
,
ν=
d − 1, 2
(46)
Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction ν 1 2κ
P∞
where Iν (·) denotes the modified Bessel function of the first kind: Iν (κ) = function.
( 14 κ2 )
k
k=0 k! Γ(ν+k+1) , Γ(·) denotes Gamma
In our model, given a normalized visual feature embedding fi ∈ Sd−1 , we parameterize each class k by vMF parameters Vk = (µk , κk ). The log-likelihood is ⊤ log pvMF (47) i,k = log Cd (κk ) + κk µk fi . Numerical approximation.
Taking logarithm of Eq. (46) yields log Cd (κ) = ν log κ −
d log(2π) − log Iν (κ). 2
(48)
Since Iν (κ) is transcendental, we adopt the large-κ asymptotic7 log Iν (κ) ≈ κ −
1 log(2πκ), 2
(49)
which leads to log Cd (κ) ≈
d−1 d−1 log κ − κ − log(2π). 2 2
(50)
In implementation, since our assignment update Eq. (12) involves a softmax over classes; terms independent of κ (e.g., 8 − d−1 2 log(2π) for fixed d) cancel out . Therefore, we use the simplified approximation form log Cd (κ) =
d−1 log κ − κ. 2
(51)
E.2. Derivation of KL divergence between two multivariate vMF distributions Consider two vMF multivariate distributions p(x) = Vp (x; µp , κp ) and q(x) = Vq (x; µq , κq ) on Sd−1 . The KullbackLeibler (KL) divergence is KL(p∥q) = Ex∼p [log p(x) − log q(x)] . (52) Using the vMF log-density from (45), we have log p(x) − log q(x) = log
Cd (κp ) ⊤ + κp µ⊤ p x − κq µq x. Cd (κq )
(53)
Taking expectation w.r.t. x ∼ p yields Cd (κp ) ⊤ KL(p∥q) = Ex∼p log + κp µ⊤ x − κ µ x q p q Cd (κq ) Cd (κp ) ⊤ = log + Ex∼p [κp µ⊤ p x] − Ex∼p [κq µq x] Cd (κq ) Cd (κp ) ⊤ = log + κp µ⊤ p Ep [x] − κq µq Ep [x]. Cd (κq )
(54) (55) (56)
A standard vMF identity states that the mean of x ∼ vMF(µp , κp ) is aligned with µp : Ep [x] = Ad (κp ) µp ,
Ad (κ) ≜
I d (κ) 2
I d −1 (κ)
,
(57)
2
7
Since there are also other tighter approximations of Iν (·), such as the uniform expansion in (Govindarajan et al., 2024) and DLMF √ P∞ Us (t) eνη (Lozier, 2003) Eq. 10.41.3: Iν (νr) ∼ √ η = 1 + r2 + log √r 2 , using our form is usually sufficient. s , 1/4 s=0 ν 2 2πν (1+r ) 1+ 1+r 8 This is because softmax function is shift invariant.
20
Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction
Substituting (57) into (56), and using ∥µp ∥2 = 1, we obtain Cd (κp ) ⊤ + κp µ⊤ p (Ad (κp )µp ) − κq µq (Ad (κp )µp ) Cd (κq ) Cd (κp ) ⊤ = log + κp Ad (κp ) (µ⊤ p µp ) − κq Ad (κp ) (µq µp ) Cd (κq ) Cd (κp ) + κp Ad (κp ) − κq Ad (κp ) µ⊤ = log q µp . Cd (κq )
KL(p∥q) = log
(58) (59) (60)
In our objective in Eq. (8), the anchor term uses KL(Vk′ ∥Vk ) with Vk′ = (µ′k , κ′k ) (anchor) and Vk = (µk , κk ) (empirical estimate). Applying (60) to p = Vk′ (µ′k , κ′k ) and q = Vk (µk , κk ) gives Cd (κ′k ) ′ + κ′k Ad (κ′k ) − κk Ad (κ′k ) µ⊤ k µk . Cd (κk )
(61)
′ log Cd (κ′k ) − log Cd (κk ) + κ′k Ad (κ′k ) − κk Ad (κ′k ) µ⊤ µ k k .
(62)
KL(Vk′ ∥Vk ) = log Therefore, the anchor term R(M) is given by R(M) =
K X k=1
When optimizing w.r.t. (µk , κk ), the terms depending on (µk , κk ) reduce to ′ − log Cd (κk ) − κk Ad (κ′k ) µ⊤ k µk ,
(63)
up to constants independent of (µk , κk ).
F. Derivation of the variable initialization of κ In Eq. (7), we initialize µ′k from the zero-shot text prototype tk , and estimate Ad (κ′k ) via a mean-squared-distance approximation on the unit hypersphere. We provide the detailed derivation of the latter in this section. Given µ′k fixed, we compute the soft, class-weighted mean squared Euclidean distance MSEk as PN MSEk ≜
′ 2 i=1 zi,k ∥fi − µk ∥2 = PN i=1 zi,k
PN
′ 2 i=1 zi,k ∥fi − µk ∥2
Nk
,
Nk ≜
N X
zi,k .
(64)
i=1
On the unit hypersphere, both fi and µ′k are ℓ2 -normalized, i.e., ∥fi ∥2 = ∥µ′k ∥2 = 1. Hence, ∥fi − µ′k ∥22 = ∥fi ∥22 + ∥µ′k ∥22 − 2(fi⊤ µ′k ) = 2 1 − cos θi ,
(65)
where cos θi ≜ fi⊤ µ′k . Taking the weighted average over i for class k gives MSEk = 2 1 − Ek [f ⊤ µ′k ] ,
Ek [f
⊤
µ′k ] ≜
zi,k fi⊤ µ′k iP
P
i zi,k
.
(66)
Therefore, Ek [f ⊤ µ′k ] = 1 −
MSEk . 2
(67)
For a vMF distribution on Sd−1 with density p(f ) = Cd (κ) exp(κµ⊤ f ), it is well-known that9 Ef ∼vMF(µ,κ) [µ⊤ f ] = Ad (κ) ≜
Id/2 (κ) , Id/2−1 (κ)
(68)
R ∂ For completeness, this follows from the partition function Z(κ) = Sd−1 exp(κµ⊤ f ) df = 1/Cd (κ) and the identity ∂κ log Z(κ) = ⊤ ′ ν E[µ f ], together with the standard Bessel recursion Iν (κ) = Iν+1 (κ) + κ Iν (κ). 9
21
Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction
where Iν (·) is the modified Bessel function of the first kind. Combining (67) and (68), we obtain the approximation Ad (κ′k ) ≈ Ek [f ⊤ µ′k ] = 1 −
MSEk =1− 2
∥fi − µ′k ∥22 i zi,kP
P
2
i zi,k
.
(69)
Finally, κ′k is approximated as the inversion of A(κk ) using Eq. (15).
G. Derivations of the variable update G.1. With respect to assignments z Following the derivations from TransCLIP (Zanella et al., 2024), we derive the update for assignments z = {zi }N i=1 under the simplex constraint zi ∈ ∆K with our instance-level, entropy-based weight γi . Note that this derivation is based on the setting in Eq. (9) that γi is computed solely from the fixed pseudo label ŷi . Although in actual implementation, we dynamically update γi with current assignment zi , we treat it more as an engineering trick. Fixing the distribution parameters (µk , κk ), the z-dependent part from the objective in Eq. (8) can be written as min
z∈(∆K )N
N X
X vMF + KL(zi ∥ŷi ) − γi ωij z⊤ γi − z ⊤ i log pi i zj .
i=1
(70)
i,j
Since ωij = fi⊤ fj ≥ 0, the affinity matrix W = [ωij ] ∈ RN ×N is positive semi-definite (PSD). This makes the Laplacian term concave with respect to z. Therefore, we adopt a BSUM-style approximation by taking the tight linear upper bound of zi at iteration t. Constructing linear upper bound. To construct such a linear upper bound, we first rewrite the Laplacian term in a matrix N ×N form. Let z ∈ RN K denote the concatenation of {zi }N be the diagonal i=1 (stacked by samples), and G = diag(γ) ∈ R matrix of instance-level weights γ. Then, the weighted Laplacian term can be written as X X ⊤ − γi ωij z⊤ (GW)ij z⊤ (71) i zj = − i zj = z Ψ z, i,j
i,j
where Ψ ≜ −(GW) ⊗ IK ,
(72)
⊗ denotes the Kronecker product, and IK is the K × K identity matrix. For notational simplicity, we use IK since each zi ∈ RK . This is also the standard lifting used in TransCLIP and StatA. When W is PSD and γ ⪰ 0, the symmetrized weight matrix 12 GW + (GW)⊤ is also PSD10 , which implies that Ψ is negative semi-definite (NSD). As a results, z⊤ Ψz is concave with respect to z. For a concave quadratic function q(z) = z⊤ Ψz with Ψ ⪯ 0, its first-order Taylor expansion at the current iterate z(t) provides a tight global upper bound: ⊤ z⊤ Ψz ≤ (z(t) )⊤ Ψz(t) + ∇q(z(t) ) (z − z(t) ),
∇q(z) = (Ψ + Ψ⊤ )z.
(73)
In particular, if Ψ is symmetric (or using its symmetric part), the gradient simplifies to ∇q(z) = 2Ψz, and the bound becomes z⊤ Ψz ≤ (z(t) )⊤ Ψz(t) + 2(Ψz(t) )⊤ (z − z(t) ). (74) This upper bound is tight in the BSUM sense, i.e., it equals the original quadratic term at z = z(t) . Moreover, replacing the quadratic coupling by (73) or (74) yields a linear surrogate that decouples across {zi } under simplex constraints, enabling an efficient BSUM update. 10
Equivalently, one may replace GW by its symmetric part without changing the scalar form of the quadratic term, since z⊤ Az =
⊤ z⊤ A+A z for any square matrix A. 2
22
Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction (t)
Therefore, by fixing the neighbors {zj }j and upper-bounding the bilinear term, we rewrite the Laplacian term as X X X X (t) (t) − γi ωij z⊤ γi ωij z⊤ + const = − γ i z⊤ ωij zj + const i zj ≈ − i zj i i,j
i,j
i
(75)
j
Substituting (75) into (70), the problem becomes separable over i. Per-sample subproblem and cancellation of γi . For each sample i, we obtain the subproblem X (t+1) (t) vMF zi ∈ arg min γi − z⊤ + KL(zi ∥ŷi ) − z⊤ ωij zj . i log pi i zi ∈∆K
(76)
j
Since γi > 0 is a constant multiplier in (76), it does not affect the minimizer: arg min γi gi (zi ) = arg min gi (zi ), zi ∈∆K
zi ∈∆K
(γi > 0),
(77)
which explains why the assignment update in Eq. (12) does not explicitly depend on γi . Closed-form update. Expanding KL(zi ∥ŷi ) = the effective subproblem is min
K X
zi ∈∆K
zi,k log zi,k −
k=1
K X
P
k zi,k log zi,k − zi,k log ŷi,k and omitting zi -independent constants,
(t)
zi,k si,k ,
(t)
si,k ≜ log ŷi,k + log pvMF i,k +
X
(t)
ωij zj,k .
(78)
j
k=1
Introducing a Lagrange multiplier λi (different from the class confidence in Eq. (11)) for constraint the Karush-Kuhn-Tucker (KKT) conditions yields (t) (t) log zi,k + 1 − si,k + λi = 0 =⇒ zi,k ∝ exp si,k .
P
k zi,k = 1, solving
(79)
Since zi ∈ ∆k , we obtain the final form after normalization P (t) ŷi,k exp log pvMF + ω z ij i,k j j,k (t+1) . zi,k = PK =P P (t) (t) K exp s ŷ exp log p + ω z i,r i,r ij i,r r=1 j,r r=1 j (t)
exp si,k
(80)
Expressed in vector form, Eq. (80) is given by (t+1) zi =
P (t) ŷi ⊙ exp(log pvMF + j ωij zj ) i . P (t) (ŷi ⊙ exp(log pvMF + j ωij zj ))⊤ 1K i
(81)
⊤ This has the same form as StatA, with log pvMF i,k = log Cd (κk ) + κk µk fi instantiated by our vMF likelihood in Eq. (6).
G.2. With respect to parameters µ and κ In this subsection, we derive the closed-form updates for the vMF parameters {µk , κk }K k=1 by fixing the soft assignments z. Recall that all feature vectors are ℓ2 -normalized, i.e., ∥fi ∥2 = 1. Update of µk . We first consider the mean direction vector µk . Collecting all terms in the PLE objective Eq. (8) that depend on µk , we obtain X ′ ⊤ ′ J(µk ) = − γi zi,k (κk µ⊤ s.t. ∥µk ∥2 = 1. (82) k fi ) + α(κk Ad (κk )µk µk ), i
Minimizing this objective is equivalent to maximizing ! max κk µ⊤ k µk
X
γi zi,k fi + αAd (κ′k )µ′k
i
23
,
s.t. ∥µk ∥2 = 1,
(83)
Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction
Let Rk ≜
X
Tk ≜ αAd (κ′k )µ′k ,
γi zi,k fi ,
(84)
i
and define the combined resultant vector Rktot ≜ Rk + Tk .
(85)
tot Then, the objective in (83) reduces to maximizing µ⊤ k Rk under a unit-norm constraint. The optimum is achieved when µk tot aligns with Rk , yielding P ′ ′ Rktot i γi zi,k fi + αAd (κk )µk P µk = . (86) = tot ′ ′ ∥Rk ∥2 ∥ i γi zi,k fi + αAd (κk )µk ∥2
Update of κk . Considering concentration parameter κk . Similarly, when fixing µk , the terms involving κk in Eq. (8) can be written as ! X L(κk ) = −(Nk + α) log Cd (κk ) − κk µ⊤ γi zi,k fi + αAd (κ′k )µ′k , (87) k i
where Nk =
P
i γi zi,k denotes the soft assignment count (with weights γi ).
From the update of µk in (86), the direction of µk coincides with that of Rktot , and thus tot tot µ⊤ k Rk = ∥Rk ∥2 .
(88)
Denoting R̄ktot ≜ ∥Rktot ∥2 , the objective (87) becomes L(κk ) = −(Nk + α) log Cd (κk ) − κk R̄ktot .
(89)
Taking the derivative with respect to κk and setting it to zero yields ∂L(κk ) ∂ ∂ = −(Nk + α) log Cd (κk ) − κk R̄ktot ∂κ ∂κ ∂κ ∂ log Cd (κk ) − R̄ktot = −(Nk + α) ∂κ = (Nk + α)Ad (κ) − R̄ktot
(90) (91) (92)
= 0,
(93)
∂ where we used the standard vMF identity ∂κ log Cd (κ) = −Ad (κ). Therefore, κk satisfies P ∥ i γi zi,k fi + αAd (κ′k )µ′k ∥2 R̄ktot P Ad (κk ) = = . Nk + α i γi zi,k + α
(94)
Then, κk is approximated as the inverse of Ad (κk ) using Eq. (15).
H. Equivalence to the Adaptive Shrinkage Form for Parameter Updates In this section, we show that the parameter updates in Eq. (13) can be written in an equivalent adaptive shrinkage form, i.e., Eq. (14). For class k, define Nk ≜
X
γi zi,k ,
Rk ≜
i
X
γi zi,k fi ,
i
vk ≜
Rk , Nk
βk ≜
Nk . Nk + α
(95)
Here, we set α as a fixed scalar for simplicity. Remark. To make the equivalence more transparent, we first present the proof under the mild simplification Ad (κ′k ) = 1, i.e., distribution anchor provides a deterministic direction (which is intuitive and natural). The same derivation holds for general Ad (κ′k ) ̸= 1 by replacing α with αAd (κ′k ). 24
Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction
With respect to µk .
Let Vkours ≜ Rk + αµ′k =
X
γi zi,k fi + αµ′k ,
VkStatA ≜ βk v k + (1 − βk )µ′k .
(96)
i
Eq. (13) updates µk by normalizing Vkours , i.e., µk =
Vkours . ∥Vkours ∥2
(97)
We now show that this is equivalent to normalizing VkStatA . Indeed, substituting the definitions in (95) yields Rk α Nk + µ′k Nk + α Nk Nk + α Rk αµ′k = + Nk + α Nk + α 1 = (Rk + αµ′k ) Nk + α 1 V ours . = Nk + α k
VkStatA =
(98) (99) (100) (101)
Therefore, VkStatA and Vkours are colinear and share the same direction. After normalization, we obtain the exact equivalence VkStatA Vkours βk v k + (1 − βk )µ′k ≡ = , ours StatA ∥Vk ∥2 ∥βk v k + (1 − βk )µ′k ∥2 ∥Vk ∥2
(102)
which proves the equivalence between Eq. (13) and (14). With respect to κk .
Eq. (13) computes Ad (κk ) =
∥V ours ∥2 ∥Rk + αµ′k ∥2 = k . Nk + α Nk + α
(103)
According to Eq. (101), we have ∥βk v k + (1 − βk )µ′k ∥2 = VkStatA 2 =
1 V ours Nk + α k 2
1 ∥V ours ∥2 Nk + α k ∥Rk + αµ′k ∥2 = Nk + α = Ad (κk ). =
(104) (105) (106) (107)
Thus, the concentration update in Eq. (13) is also exactly equivalent to the form in Eq. (14).
I. Additional Experimental Results I.1. Results with different batch sizes To verify the robustness of MOON concerning data availability, we conduct experiments with varying batch sizes as reported in Tab. 11. Across three different batch sizes and two realistic scenarios, our method consistently outperforms all baselines and significantly improves upon the zero-shot CLIP initialization. This indicates the effectiveness of MOON, regardless of whether the incoming data batch is sparse or abundant, ensuring reliable adaptation performance across different data scales.
25
Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction
Table 11. Results with different batch sizes. The best and second-best results are marked in bold and underlined, respectively. Subscript green indicates improvement, red indicates decline, and gray indicates no change compared with zero-shot performance.
CF 10 U
66.6
62.5
24.7
48.3
65.6
85.9
89.1
70.7
93.2
43.5
67.5
65.2
Medium
StatA MOON
72.0+5.4 82.3+15.7
66.5+4.0 76.2+13.7
26.6+1.9 27.1+2.4
48.2−0.1 46.2−2.1
72.7+7.1 75.7+10.1
88.7+2.8 93.8+7.9
92.1+3.0 92.1+3.0
75.4+4.7 75.9+5.2
93.7+0.5 94.8+1.6
47.1+4.2 47.1+3.6
70.0+2.5 72.9+5.4
68.5+3.3 71.3+6.1
High
StatA MOON
69.4+2.8 76.4+9.8
64.9+2.4 69.6+7.1
23.6−1.1 21.7−3.0
47.2−1.1 46.2−2.1
68.0+2.4 68.6+3.0
87.0+1.1 88.9+3.0
88.2−0.9 86.4−2.7
72.0+1.3 71.3+0.6
94.0+0.8 93.7+0.5
46.9+3.4 40.2−3.3
68.2+0.7 67.1−0.4
66.3+1.1 66.4+1.1
U CF
Avg.
CLIP
TD
D
TD
1
ch 1
01
2 Pe
Ca lte
er s1 0 ts
d1 Fo o
St
Fl ow
01
dC or an f
SA T Eu
ro
ft ra
39
A irc
Method
Im
Keff
SU N
ag
eN
7
et
ar s
(a) Batch Size: 128.
10
1 D
Ca lte
ch 1
01
02 er s1 Fl ow
St
Fo o
Pe ts
d1 0
1
or dC an f
oS AT
irc
Eu r
ra ft
39 N
A
Method
SU
Keff
Im
ag e
N
7
et
ar
s
(b) Batch Size: 256.
Avg.
CLIP
66.6
62.5
24.7
48.3
65.6
85.9
89.1
70.7
93.2
43.5
67.5
65.2
Medium
StatA MOON
72.0+5.4 82.9+16.3
66.7+4.2 77.8+15.3
27.1+2.4 27.9+3.2
56.0+7.7 49.4+1.1
74.1+8.5 76.6+11.0
88.9+3.0 94.1+8.2
92.9+3.8 92.9+3.8
76.0+5.3 76.1+5.4
93.6+0.4 95.2+2.0
47.0+3.5 48.4+4.9
70.5+3.0 73.7+6.2
69.5+4.3 72.3+7.0
High
StatA MOON
71.1+4.5 80.5+13.9
66.3+3.8 73.2+10.7
24.2−0.5 23.2−1.5
55.5+7.2 49.1+0.8
70.6+5.0 71.8+6.2
87.6+1.7 90.4+4.5
88.9−0.2 88.2−0.9
73.7+3.0 73.1+2.4
94.1+0.9 94.1+0.9
47.0+3.5 42.7−0.8
69.9+2.4 69.6+2.1
68.1+2.9 68.7+3.5
10 CF U
TD D
Ca
lte
ch
1
10
1
02 er s1 Fl ow
Pe
d1 Fo o
ts
01
Ca rd fo an St
AT Eu
ra irc
ro S
ft
7 N
39 A
Method
SU
Keff
Im
ag eN
et
rs
(c) Batch Size: 500.
Avg.
CLIP
66.6
62.5
24.7
48.3
65.6
85.9
89.1
70.7
93.2
43.5
67.5
65.2
Medium
StatA MOON
71.5+4.9 82.4+15.8
65.5+3.0 76.7+14.2
27.8+3.1 28.6+3.9
59.3+11.0 55.5+7.2
74.9+9.3 77.0+11.4
88.3+2.4 93.7+7.8
93.1+4.0 92.7+3.6
76.8+6.1 76.5+5.8
93.1−0.1 95.0+1.8
47.1+3.6 49.0+5.5
69.9+2.4 73.4+5.9
69.8+4.5 72.8+7.5
High
StatA MOON
72.1+5.5 82.3+15.7
67.3+4.8 75.5+13.0
25.1+0.4 24.3−0.4
60.0+11.7 55.8+7.5
72.3+6.7 73.8+8.2
88.2+2.3 91.2+5.3
90.3+1.2 89.3+0.2
75.5+4.8 74.5+3.8
93.8+0.6 94.7+1.5
47.2+3.7 44.1+0.6
70.7+3.2 70.8+3.3
69.3+4.1 70.6+5.3
26
Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction Table 12. Overview of the VLMs employed in our experiments. Model name
Full name
#Param
Detailed #Param
FLOPS (B)
Resolution
Pretrained
CLIP CLIP CLIP CLIP CLIP OpenCLIP SigLIP EVA-CLIP
clip-resnet50 / RN50 clip-resnet101 / RN101 clip-vit-base-patch16 / ViT-B/16 clip-vit-base-patch32 / ViT-B/32 clip-vit-large-patch14 / ViT-L/14 ViT-B-16 ViT-SO400M-14-SigLIP-384 EVA01-g-14
102M 120M 150M 151M 428M 150M 878M 1.14B
102,007,137 119,688,033 149,620,737 151,277,313 427,616,513 149,620,737 877,960,498 1,136,435,841
18.18 25.50 41.09 14.78 175.33 41.09 723.48 547.36
224x224 224x224 224x224 224x224 224x224 224x224 384x384 224x224
WIT WIT WIT WIT WIT DataComp WebLI LAION
I.2. Results with other backbones Following the discussion in Sec. 5.3, we extend our evaluation to four additional CLIP visual backbones, including ResNet (He et al., 2016) architectures (RN50, RN101) and Vision Transformers (ViT) (Dosovitskiy et al., 2021) of varying scales (ViT-B/32, ViT-L/14), as detailed in Tab. 13, 14 and 15. These experiments cover the full range of realistic batch and online adaptation settings. MOON demonstrates remarkable universality, achieving the highest accuracy in nearly all evaluated scenarios (winning in 33 out of 36 cases) with only negligible margins in the few exceptions. This consistent superiority across diverse architectures and model capacities confirms that our vMF-based dynamic shrinkage mechanism is a generalized solution, capable of enhancing VLM performance regardless of the specific underlying visual encoder. I.3. Results with other VLM architectures To validate the generalizability of our MOON beyond standard CLIP models, we evaluate three additional VLMs with diverse architectures, parameter scales, and training procedures: OpenCLIP (151M) (Cherti et al., 2023), SigLIP (878M) (Zhai et al., 2023), and EVA-CLIP (1.1B) (Sun et al., 2023), all implemented using the OpenCLIP codebase (Ilharco et al., 2021). Details regarding the VLMs in our experiment is presented in Tab. 12. Experiments conducted with a batch size of 1,000 across three realistic scenarios (Tables 16, 17, and 18) reveal that MOON consistently enhances zero-shot performance and achieves state-of-the-art results in nearly all nine settings. Notably, the relative improvements are particularly distinct for smaller models (e.g., OpenCLIP), suggesting our method effectively compensates for weaker initial representations, while maintaining substantial gains on challenging large-scale datasets like ImageNet and SUN397. Collectively, these findings demonstrate the universal effectiveness of MOON across diverse model architectures, scales, and pre-training paradigms. I.4. Results on full dataset with all classes We further evaluate the extreme scenario where the model adapts to the full dataset containing all classes simultaneously, representing a dense label distribution that deviates from our sparsity assumption. As shown in Tab. 19, MOON trails the strongest baseline StatA by a marginal gap (∼2%). This behavior is consistent with our analysis in the main text: our dynamic shrinkage mechanism is inherently designed with an inductive bias to favor sparse effective class sets, which is less optimal when the ground-truth distribution is uniform. However, unlike many specialized methods that collapse when assumptions are violated, MOON maintains competitive high-accuracy performance without severe degradation. Considering its significant computational efficiency, our method offers a robust trade-off, serving as a reliable solution even in scenarios with maximal class presence. I.5. Results on random class scenarios To simulate a highly unpredictable deployment environment, we introduce a challenging ”Random” scenario where the number of effective classes Kef f varies stochastically between 1 and min{N, K} for each adaptation step. This setting effectively aggregates the characteristics of varying sparsity levels into a single dynamic evaluation, with results demonstrated in Tab. 20. Under these volatile conditions, MOON exhibits remarkable stability, achieving performance comparable to the state-of-the-art StatA across varying batch sizes. The fact that our method matches the strongest baseline in accuracy while operating with significantly lower latency and a simpler optimization procedure (as detailed in Appendix A) further underscores its practicality for handling real-world data streams with unknown and fluctuating statistics. 27
Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction
Table 13. Results on four additional CLIP backbones, batch adaptation with batch size of 64. Subscript green indicates improvement, red indicates decline, and gray indicates no change compared with zero-shot performance.
U CF 10
55.8
77.4
85.7
66.1
85.7
42.8
61.8
58.7
Very Low
StatA MOON
68.2+10.0 79.7+21.5
63.7+4.8 77.5+18.6
21.1+4.1 25.4+8.4
43.3+7.1 41.8+5.6
71.3+15.5 74.8+19.0
87.1+9.7 94.9+17.5
93.1+7.4 95.5+9.8
74.1+8.0 75.2+9.1
90.1+4.4 92.2+6.5
45.3+2.5 54.4+11.6
66.2+4.4 74.1+12.3
65.8+7.1 71.4+12.7
Low
StatA MOON
65.0+6.8 79.5+21.3
62.8+3.9 77.9+19.0
17.8+0.8 21.9+4.9
31.7−4.5 35.2−1.0
67.1+11.3 72.8+17.0
83.6+6.2 92.9+15.5
88.2+2.5 94.0+8.3
71.7+5.6 75.6+9.5
89.0+3.3 91.7+6.0
44.9+2.1 49.2+6.4
64.0+2.2 71.6+9.8
62.3+3.7 69.3+10.6
Medium
StatA MOON
61.1+2.9 73.6+15.4
59.5+0.6 71.3+12.4
16.4−0.6 18.6+1.6
27.3−8.9 31.9−4.3
62.1+6.3 67.2+11.4
78.7+1.3 87.6+10.2
81.4−4.3 88.5+2.8
64.1−2.0 70.5+4.4
87.9+2.2 89.8+4.1
43.8+1.0 42.2−0.6
62.0+0.2 67.5+5.7
58.6−0.1 64.4+5.7
Ca
Fl
Pe
Fo o
A
Eu
N SU
ow
36.2
ts
17.0
St an f
58.9
ro S
58.2
Method
irc r
CLIP
Keff
Im ag
D TD
1
10 1 lte ch
er s1 02
d1 01
or dC
AT
af t
39
eN
7
et
ar s
(a) ResNet-50.
Avg.
01
2
59.0
17.9
32.9
63.2
80.7
86.9
64.3
89.9
37.3
61.1
59.5
Very Low
StatA MOON
73.0+11.7 80.0+18.7
66.5+7.5 74.5+15.5
22.5+4.6 24.8+6.9
30.4−2.5 30.8−2.1
76.2+13.0 77.9+14.7
89.5+8.8 94.1+13.4
95.2+8.3 93.9+7.0
74.6+10.3 72.4+8.1
91.9+2.0 93.2+3.3
42.9+5.6 46.1+8.8
65.1+4.0 70.3+9.2
66.2+6.7 68.9+9.4
Low
StatA MOON
71.2+9.9 79.8+18.5
65.9+6.9 74.5+15.5
20.0+2.1 21.8+3.9
29.6−3.3 30.0−2.9
73.1+9.9 76.4+13.2
88.1+7.4 93.1+12.4
92.9+6.0 92.5+5.6
74.9+10.6 72.6+8.3
92.8+2.9 94.0+4.1
42.9+5.6 42.6+5.3
64.4+3.3 68.4+7.3
65.1+5.6 67.8+8.3
Medium
StatA MOON
67.0+5.7 74.5+13.2
62.7+3.7 69.1+10.1
18.6+0.7 19.7+1.8
28.7−4.2 29.9−3.0
69.6+6.4 72.7+9.5
84.9+4.2 89.0+8.3
88.8+1.9 89.0+2.1
70.1+5.8 68.2+3.9
92.1+2.2 92.6+2.7
40.9+3.6 37.5+0.2
63.6+2.5 65.3+4.2
62.5+3.0 64.3+4.8
1 CF 10 U
TD D
Ca l
ow Fl
ts
fo r an St
ft A
SU
N
39
eN ag Im
Method
te ch 1
Pe
er s1 0
Fo od 10
61.3
Keff
1
Eu ro SA T
dC a
ra
CLIP
7
irc
et
rs
(b) ResNet-101.
Avg.
10 CF
Avg.
U
D
Ca
TD
lte
ch
1
10
1
02 er s1 Pe
Fl ow
d1 0
ts
1
Ca rd fo
Fo o
Eu
St an
SA T
ra irc
ro
ft
7 39 N
A
Method
SU
Keff
Im ag
eN et
rs
(c) ViT-B/32.
CLIP
62.0
62.1
19.1
45.4
60.2
80.4
87.3
66.6
91.4
42.7
63.5
61.9
Very Low
StatA MOON
68.1+6.1 80.9+18.9
65.6+3.5 78.4+16.3
23.0+3.9 25.7+6.6
53.2+7.8 52.2+6.8
71.9+11.7 77.8+17.6
85.8+5.4 94.8+14.4
94.3+7.0 94.7+7.4
74.9+8.3 75.4+8.8
93.4+2.0 95.1+3.7
45.2+2.5 53.6+10.9
64.5+1.0 70.4+6.9
67.3+5.4 72.6+10.7
Low
StatA MOON
67.2+5.2 80.3+18.3
65.7+3.6 78.8+16.7
21.9+2.8 23.8+4.7
50.1+4.7 45.6+0.2
69.3+9.1 77.0+16.8
84.5+4.1 93.7+13.3
92.5+5.2 93.1+5.8
75.3+8.7 76.4+9.8
93.2+1.8 94.8+3.4
46.1+3.4 47.9+5.2
64.6+1.1 69.6+6.1
66.4+4.5 71.0+9.1
Medium
StatA MOON
65.5+3.5 75.6+13.6
64.8+2.7 73.6+11.5
20.0+0.9 20.3+1.2
45.4±0.0 40.8−4.6
65.0+4.8 70.8+10.6
82.6+2.2 89.1+8.7
89.4+2.1 89.6+2.3
70.6+4.0 71.1+4.5
92.9+1.5 93.6+2.2
46.7+4.0 42.0−0.7
64.4+0.9 67.1+3.6
64.3+2.4 66.7+4.8
1 U CF
10
D TD
93.5
79.5
95.2
53.5
74.9
72.6
Very Low
StatA MOON
78.9+5.4 85.8+12.3
71.3+3.6 80.1+12.4
40.4+7.9 41.3+8.8
71.4+11.1 72.4+12.1
84.4+7.5 86.2+9.3
94.2+3.3 97.9+7.0
97.1+3.6 97.5+4.0
82.9+3.4 83.5+4.0
97.0+1.8 98.1+2.9
55.3+1.8 64.7+11.2
77.1+2.2 80.8+5.9
77.3+4.7 80.8+8.2
Low
StatA MOON
78.2+4.7 86.4+12.9
71.6+3.9 80.8+13.1
38.4+5.9 39.7+7.2
65.6+5.3 65.2+4.9
82.4+5.5 85.2+8.3
93.1+2.2 97.2+6.3
96.3+2.8 97.5+4.0
82.8+3.3 83.7+4.2
96.1+0.9 97.7+2.5
55.4+1.9 60.7+7.2
76.8+1.9 79.7+4.8
76.1+3.5 79.4+6.8
Medium
StatA MOON
76.6+3.1 83.3+9.8
70.0+2.3 76.3+8.6
36.4+3.9 36.3+3.8
62.6+2.3 61.4+1.1
80.6+3.7 83.2+6.3
92.1+1.2 95.4+4.5
93.9+0.4 94.9+1.4
80.8+1.3 81.5+2.0
95.6+0.4 96.7+1.5
54.6+1.1 54.5+1.0
77.1+2.2 78.2+3.3
74.5+2.0 76.5+3.9
Pe t
Fo o
Eu
Im
St
28
Ca
90.9
Fl
76.9
s
60.3
an
32.5
ro
67.7
Method
A irc
73.5
Keff
SU N
CLIP
ag
lte
ch
s1
10
1
02 d1
ow er
01
Ca rd fo
SA T
ft ra
7 39
eN
et
rs
(d) ViT-L/14.
Avg.
Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction
Table 14. Results on four additional CLIP backbones, batch adaptation with batch size of 1,000. Subscript green indicates improvement, red indicates decline, and gray indicates no change compared with zero-shot performance.
85.7
42.8
61.8
58.7
Medium
StatA MOON
65.2+7.0 78.2+20.0
61.5+2.6 75.1+16.2
18.6+1.6 21.3+4.3
51.2+15.0 43.3+7.1
67.2+11.4 70.6+14.8
80.9+3.5 87.5+10.1
89.1+3.4 89.7+4.0
70.7+4.6 73.4+7.3
88.5+2.8 91.1+5.4
46.8+4.0 47.8+5.0
65.3+3.5 72.2+10.4
64.1+5.4 68.2+9.5
High
StatA MOON
65.4+7.2 77.5+19.3
63.1+4.2 74.1+15.2
16.5−0.5 18.1+1.1
51.7+15.5 43.2+7.0
65.4+9.6 67.5+11.7
81.0+3.6 84.3+6.9
84.4−1.3 86.5+0.8
70.0+3.9 71.3+5.2
88.3+2.6 89.3+3.6
47.2+4.4 43.6+0.8
66.0+4.2 69.2+7.4
63.5+4.8 65.9+7.2
Very High
StatA MOON
63.5+5.3 74.7+16.5
62.4+3.5 70.5+11.6
14.8−2.2 16.7−0.3
51.7+15.5 43.2+7.0
60.8+5.0 62.3+6.5
77.8+0.4 79.6+2.2
83.5−2.2 85.9+0.2
66.2+0.1 68.3+2.2
87.9+2.2 86.1+0.4
46.6+3.8 42.5−0.3
64.5+2.7 64.8+3.0
61.8+3.1 63.1+4.4
CF
Avg.
U
D
N SU
Im
10 1
66.1
TD
85.7
Ca lte ch 1
77.4
Fl
01
2
55.8
Pe ts
er s1 0 Pe ts
36.2
Fl ow
Fo od 10 1
an fo rd Ca
17.0
St
Eu ro SA T
58.9
Method
ft ra
58.2
Keff
39 7
CLIP
ag eN
A irc
et
rs
(a) ResNet-50.
CF 10 1 U
D
TD
01 Ca lte ch 1
10 2 ow
er s
1 Fo o
d1 0
rd Ca rs fo St an
SA T Eu ro
ft irc ra
39 N
A
Method
SU
Keff
Im
ag e
N
7
et
(b) ResNet-101.
Avg.
CLIP
61.3
59.0
17.9
32.9
63.2
80.7
86.9
64.3
89.9
37.3
61.1
59.5
Medium
StatA MOON
70.5+9.2 77.6+16.3
65.3+6.3 72.6+13.6
20.5+2.6 21.5+3.6
33.6+0.7 33.3+0.4
73.9+10.7 75.3+12.1
85.4+4.7 88.9+8.2
91.1+4.2 89.6+2.7
73.1+8.8 71.2+6.9
92.2+2.3 94.2+4.3
43.2+5.9 41.6+4.3
66.5+5.4 68.7+7.6
65.0+5.5 66.8+7.3
High
StatA MOON
71.4+10.1 77.7+16.4
66.2+7.2 71.2+12.2
18.6+0.7 19.2+1.3
32.8−0.1 33.1+0.2
72.2+9.0 72.6+9.4
85.1+4.4 86.3+5.6
87.9+1.0 87.6+0.7
71.9+7.6 69.1+4.8
92.2+2.3 93.1+3.2
42.5+5.2 38.7+1.4
66.5+5.4 65.9+4.8
64.3+4.8 65.0+5.5
Very High
StatA MOON
70.1+8.8 75.9+14.6
65.4+6.4 67.9+8.9
16.9−1.0 17.9±0.0
32.9±0.0 33.1+0.2
68.2+5.0 68.7+5.5
82.4+1.7 82.7+2.0
87.2+0.3 87.1+0.2
68.7+4.4 65.8+1.5
91.3+1.4 90.2+0.3
41.9+4.6 37.7+0.4
63.8+2.7 62.2+1.1
62.6+3.1 62.7+3.2
CF 10 U
TD D
Ca
lte
ch
1
10
1
02 s1 ow Fl
ts Pe
d1 Fo o
er
01
Ca rd fo an St
SA T Eu
ro
ft
7 39
ra A irc
Method
SU N
Keff
Im
ag
eN et
rs
(c) ViT-B/32.
Avg.
CLIP
62.0
62.1
19.1
45.4
60.2
80.4
87.3
66.6
91.4
42.7
63.5
61.9
Medium
StatA MOON
65.9+3.9 79.1+17.1
63.3+1.2 76.0+13.9
21.9+2.8 22.7+3.6
51.3+5.9 52.1+6.7
69.3+9.1 74.4+14.2
82.2+1.8 89.1+8.7
90.3+3.0 90.3+3.0
74.1+7.5 74.0+7.4
92.6+1.2 94.2+2.8
47.4+4.7 47.0+4.3
66.1+2.6 69.2+5.7
65.9+4.0 69.8+7.9
High
StatA MOON
67.0+5.0 79.3+17.3
65.0+2.9 75.1+13.0
20.2+1.1 20.2+1.1
51.1+5.7 52.1+6.7
68.5+8.3 71.6+11.4
82.7+2.3 86.5+6.1
88.5+1.2 87.9+0.6
73.7+7.1 71.8+5.2
92.5+1.1 93.7+2.3
49.5+6.8 43.8+1.1
66.9+3.4 67.9+4.4
66.0+4.1 68.2+6.3
Very High
StatA MOON
66.6+4.6 76.9+14.9
66.0+3.9 72.3+10.2
18.8−0.3 18.4−0.7
51.0+5.6 52.1+6.7
65.1+4.9 66.9+6.7
81.5+1.1 82.6+2.2
88.0+0.7 87.5+0.2
70.6+4.0 68.6+2.0
91.9+0.5 91.6+0.2
49.5+6.8 43.0+0.3
66.5+3.0 65.3+1.8
65.1+3.2 65.9+4.0
79.5
95.2
53.5
74.9
72.6
Medium
StatA MOON
76.2+2.7 85.0+11.5
69.4+1.7 79.2+11.5
39.1+6.6 38.9+6.4
71.0+10.7 67.4+7.1
81.9+5.0 84.0+7.1
91.7+0.8 95.9+5.0
94.8+1.3 95.4+1.9
81.9+2.4 83.3+3.8
95.6+0.4 97.7+2.5
56.9+3.4 59.4+5.9
77.6+2.7 80.2+5.3
76.0+3.4 78.8+6.2
High
StatA MOON
77.2+3.7 85.3+11.8
70.9+3.2 78.3+10.6
36.8+4.3 35.3+2.8
71.2+10.9 67.2+6.9
82.0+5.1 83.6+6.7
92.3+1.4 94.6+3.7
94.3+0.8 94.2+0.7
81.9+2.4 82.5+3.0
95.3+0.1 96.9+1.7
58.7+5.2 55.9+2.4
78.8+3.9 79.0+4.1
76.3+3.7 77.5+5.0
Very High
StatA MOON
77.3+3.8 84.5+11.0
71.6+3.9 76.0+8.3
33.7+1.2 32.6+0.1
71.2+10.9 67.2+6.9
79.5+2.6 80.9+4.0
91.7+0.8 92.4+1.5
94.1+0.6 94.0+0.5
80.7+1.2 80.3+0.8
94.9−0.3 95.1−0.1
59.0+5.5 54.8+1.3
78.7+3.8 76.6+1.7
75.7+3.1 75.9+3.3
29
U
D
Ca
Fl
Eu
A irc ra
SU N
ag e Im
CF 10 1
93.5
TD
Pe ts
90.9
lte
Fo od 10 1
76.9
ow er
rd C
60.3
St an
32.5
ro SA T
67.7
Method
ft
73.5
Keff
39 7
CLIP
N et
fo
ch 10 1
s1 02
ar s
(d) ViT-L/14.
Avg.
Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction
Table 15. Results on four additional CLIP backbones, online adaptation with batch size of 128. Subscript green indicates improvement, red indicates decline, and gray indicates no change compared with zero-shot performance.
CF 10 U
D
Ca
TD
lte
ch
1
10
1
2 er s1 0 Fl ow
Pe t
St
Fo o
s
d1
01
or dC an f
SA T Eu
ro
ft A
irc ra
39 SU
Method
Im
Scenario
N
ag
eN
7
et
ar s
(a) ResNet-50.
Avg.
CLIP
58.2
58.9
17.0
36.2
55.8
77.4
85.7
66.1
85.7
42.8
61.8
58.7
Low
StatA MOON
54.6−3.6 58.2±0.0
56.6−2.3 60.4+1.5
15.1−1.9 16.2−0.8
39.7+3.5 39.5+3.3
57.6+1.8 57.7+1.9
79.4+2.0 85.2+7.8
85.1−0.6 89.5+3.8
60.7−5.4 66.7+0.6
87.8+2.1 87.8+2.1
44.4+1.6 43.0+0.2
61.7−0.1 62.9+1.1
58.4−0.3 60.6+1.9
Medium
StatA MOON
59.6+1.4 70.3+12.1
60.8+1.9 72.9+14.0
17.7+0.7 20.9+3.9
43.5+7.3 41.0+4.8
65.9+10.1 69.7+13.9
84.5+7.1 92.5+15.1
90.6+4.9 93.7+8.0
68.1+2.0 72.0+5.9
89.3+3.6 90.5+4.8
45.5+2.7 48.3+5.5
64.5+2.7 70.6+8.8
62.8+4.1 67.5+8.8
High
StatA MOON
64.7+6.5 78.3+20.1
62.6+3.7 77.1+18.2
18.5+1.5 22.2+5.2
43.6+7.4 41.3+5.1
68.5+12.7 72.9+17.1
85.8+8.4 93.4+16.0
92.2+6.5 94.2+8.5
70.1+4.0 73.6+7.5
89.8+4.1 91.4+5.7
45.9+3.1 49.7+6.9
65.2+3.4 72.8+11.0
64.3+5.6 69.7+11.0
Separate
StatA MOON
66.6+8.4 79.4+21.2
62.6+3.7 76.7+17.8
19.8+2.8 24.7+7.7
44.3+8.1 40.3+4.1
69.5+13.7 73.3+17.5
85.6+8.2 92.8+15.4
93.8+8.1 94.1+8.4
71.9+5.8 74.9+8.8
90.2+4.5 92.3+6.6
46.0+3.2 53.0+10.2
65.3+3.5 73.8+12.0
65.1+6.4 70.5+11.8
01
2 er s1 0
32.9
63.2
80.7
86.9
64.3
89.9
37.3
61.1
59.5
Low
StatA MOON
60.5−0.8 61.8+0.5
59.3+0.3 60.3+1.3
16.9−1.0 17.6−0.3
32.7−0.2 32.4−0.5
65.5+2.3 65.9+2.7
84.9+4.2 87.5+6.8
91.0+4.1 89.9+3.0
67.8+3.5 65.7+1.4
92.2+2.3 91.8+1.9
41.1+3.8 37.7+0.4
62.8+1.7 61.9+0.8
61.3+1.8 61.1+1.6
Medium
StatA MOON
66.1+4.8 71.9+10.6
64.2+5.2 70.5+11.5
19.7+1.8 21.0+3.1
33.3+0.4 33.0+0.1
72.2+9.0 74.1+10.9
88.1+7.4 92.6+11.9
94.1+7.2 92.4+5.5
72.1+7.8 69.6+5.3
93.2+3.3 93.9+4.0
42.9+5.6 41.4+4.1
65.2+4.1 67.8+6.7
64.6+5.1 66.2+6.7
High
StatA MOON
70.5+9.2 78.2+16.9
65.9+6.9 74.1+15.1
20.6+2.7 21.8+3.9
33.5+0.6 33.4+0.5
74.1+10.9 76.3+13.1
88.7+8.0 93.2+12.5
94.4+7.5 92.8+5.9
73.1+8.8 70.8+6.5
93.4+3.5 94.6+4.7
43.0+5.7 42.5+5.2
65.7+4.6 69.5+8.4
65.7+6.2 67.9+8.4
Separate
StatA MOON
71.4+10.1 78.8+17.5
65.7+6.7 74.0+15.0
22.1+4.2 23.8+5.9
32.2−0.7 33.8+0.9
74.9+11.7 76.8+13.6
88.5+7.8 92.7+12.0
94.2+7.3 92.7+5.8
73.9+9.6 72.1+7.8
93.4+3.5 95.2+5.3
41.9+4.6 44.1+6.8
65.7+4.6 70.3+9.2
65.8+6.3 68.6+9.1
1 10 CF
Avg.
U
D TD
lte Ca
Pe
Fo o
Fl
rd fo an St
ft A
SU
N
39
eN ag Im
ch 1
17.9
ow
59.0
Method
ts
61.3
Scenario
d1 01
Eu ro SA T
Ca
ra
CLIP
7
irc
et
rs
(b) ResNet-101.
CF 10 U
D
Ca
TD
lte
ch
1
10 1
02 er s1 Pe
Fl ow
d1 Fo o
St an
ts
01
Ca fo rd
SA T Eu ro
irc
ra
ft
39 7 N
A
Method
SU
Scenario
Im
ag eN
et
rs
(c) ViT-B/32.
Avg.
CLIP
62.0
62.1
19.1
45.4
60.2
80.4
87.3
66.6
91.4
42.7
63.5
61.9
Low
StatA MOON
61.4−0.6 62.4+0.4
62.7+0.6 63.6+1.5
19.2+0.1 18.2−0.9
51.0+5.6 49.9+4.5
61.8+1.6 62.2+2.0
82.6+2.2 87.4+7.0
91.0+3.7 90.3+3.0
69.0+2.4 67.8+1.2
92.9+1.5 92.6+1.2
46.4+3.7 42.3−0.4
64.4+0.9 64.1+0.6
63.9+2.0 63.7+1.8
Medium
StatA MOON
64.6+2.6 72.9+10.9
64.8+2.7 74.3+12.2
21.4+2.3 22.2+3.1
49.9+4.5 50.7+5.3
68.1+7.9 73.7+13.5
84.4+4.0 93.1+12.7
92.8+5.5 93.1+5.8
72.5+5.9 72.2+5.6
93.5+2.1 94.2+2.8
46.4+3.7 47.4+4.7
65.5+2.0 69.2+5.7
65.8+3.9 69.4+7.5
High
StatA MOON
66.9+4.9 79.7+17.7
64.9+2.8 77.7+15.6
22.0+2.9 23.4+4.3
50.1+4.7 51.1+5.7
69.9+9.7 76.6+16.4
84.6+4.2 93.8+13.4
93.2+5.9 93.6+6.3
73.5+6.9 73.6+7.0
93.7+2.3 94.7+3.3
46.3+3.6 48.4+5.7
65.6+2.1 70.4+6.9
66.4+4.6 71.2+9.3
Separate
StatA MOON
67.0+5.0 80.3+18.3
63.8+1.7 77.3+15.2
22.9+3.8 25.4+6.3
44.9−0.5 49.1+3.7
70.4+10.2 77.0+16.8
84.1+3.7 93.1+12.7
92.8+5.5 93.4+6.1
74.6+8.0 75.0+8.4
94.0+2.6 95.1+3.7
45.1+2.4 50.6+7.9
65.0+1.5 70.4+6.9
65.9+4.0 71.5+9.6
CF 10
TD
79.5
95.2
53.5
74.9
72.6
Low
StatA MOON
73.3−0.2 73.8+0.3
68.2+0.5 68.3+0.6
34.1+1.6 32.2−0.3
68.8+8.5 67.8+7.5
77.7+0.8 77.7+0.8
92.0+1.1 94.5+3.6
95.0+1.5 95.2+1.7
80.2+0.7 79.6+0.1
95.6+0.4 96.0+0.8
55.4+1.9 54.0+0.5
76.9+2.0 75.9+1.0
74.3+1.7 74.1+1.5
Medium
StatA MOON
75.8+2.3 81.3+7.8
70.6+2.9 77.3+9.6
38.3+5.8 37.9+5.4
69.8+9.6 69.0+8.7
81.9+5.0 83.9+7.0
93.2+2.3 97.2+6.3
96.3+2.8 96.8+3.3
81.5+2.0 82.0+2.5
95.8+0.6 97.1+1.9
55.6+2.1 58.8+5.3
77.6+2.7 79.6+4.7
76.0+3.4 78.3+5.7
High
StatA MOON
77.6+4.1 85.6+12.1
71.1+3.4 80.3+12.6
39.6+7.1 39.7+7.2
68.9+8.6 69.4+9.1
82.9+6.0 85.2+8.3
93.5+2.6 97.5+6.6
96.5+3.0 97.0+3.5
81.9+2.4 82.6+3.1
95.7+0.5 97.4+2.2
55.5+2.0 60.3+6.8
77.5+2.6 80.4+5.5
76.4+3.8 79.6+7.0
Separate
StatA MOON
77.6+4.1 85.7+12.2
70.5+2.8 80.4+12.7
41.3+8.8 41.4+8.9
66.3+6.0 68.1+7.8
83.2+6.3 85.2+8.3
93.5+2.6 97.4+6.5
96.3+2.8 96.8+3.3
82.0+2.5 83.4+3.9
95.8+0.6 97.7+2.5
54.5+1.0 62.6+9.1
76.8+1.9 80.5+5.6
76.1+3.6 79.9+7.3
30
U
93.5
D
90.9
Ca
76.9
Fl ow
60.3
ts Pe
32.5
St
67.7
Eu
Fo od
lte
ch
1
10
1
02 er s1
10
1
rd C an fo
AT ro S
ft ra A irc
39
73.5
Method
SU
CLIP
Scenario
Im
N
ag
eN
7
et
ar s
(d) ViT-L/14.
Avg.
Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction
89.9
87.5
92.8
75.4
96.7
58.3
67.5
72.5
CF 10 U
TD D
Eu r
St an fo
t ra f A irc
SU N
1
Ca lte ch 1
01
2 Fl ow er
s1 0 Pe ts
56.4
10 1 Fo od
29.7
ag eN
Method OpenCLIP
rd Ca rs
69.9
39 7
73.0
Im
Keff
oS AT
et
Table 16. Results on OpenCLIP (151M), batch adaptation with batch size of 1,000. Subscript green indicates improvement, red indicates decline, and gray indicates no change compared with zero-shot performance.
Avg.
Medium
StatA MOON
73.4+0.4 83.5+10.5
68.8−1.1 80.0+10.1
32.6+2.8 32.6+2.9
62.3+5.9 61.5+5.1
91.2+1.3 93.9+4.0
87.4−0.2 93.3+5.8
93.9+1.0 95.3+2.5
79.7+4.2 78.4+3.0
96.4−0.3 98.1+1.4
59.7+1.4 65.1+6.8
70.2+2.7 74.0+6.6
74.1+1.7 77.8+5.3
High
StatA MOON
74.1+1.1 83.7+10.7
70.1+0.3 80.2+10.4
32.1+2.4 31.0+1.2
62.4+6.0 61.5+5.1
91.4+1.5 93.9+4.0
88.2+0.7 91.7+4.2
93.7+0.9 93.7+0.8
79.7+4.3 77.5+2.0
96.7+0.1 97.9+1.2
62.1+3.7 61.3+2.9
71.1+3.6 72.3+4.8
74.7+2.2 76.8+4.3
Very High
StatA MOON
74.6+1.6 83.1+10.1
71.3+1.4 79.1+9.2
30.9+1.1 28.8−0.9
62.4+6.0 61.4+5.0
91.2+1.3 93.1+3.2
88.2+0.7 89.3+1.7
93.6+0.8 93.4+0.6
77.8+2.3 75.6+0.1
96.7±0.0 96.6−0.1
62.6+4.3 60.5+2.1
71.4+3.9 70.4+2.9
74.6+2.1 75.6+3.1
1
2
1 CF 10 U
D
TD
Ca lte
ch 10
10 ow Fl
Pe
Fo o
ts
d1
er s
01
Ca
Eu
St an
ro
fo
rd
SA T
ft ra irc
Method
A
SU
Im
Keff
N
ag
39
eN
7
et
rs
Table 17. Results on SigLIP (878M), batch adaptation with batch size of 1,000. Subscript green indicates improvement, red indicates decline, and gray indicates no change compared with zero-shot performance.
Avg.
SigLIP
82.3
75.4
60.2
57.1
94.7
94.7
96.5
92.7
98.2
64.8
83.7
81.8
Medium
StatA MOON
82.7+0.5 91.4+9.1
76.1+0.7 86.0+10.6
64.7+4.5 65.8+5.6
60.4+3.3 59.4+2.3
95.4+0.7 96.8+2.1
94.0−0.7 98.1+3.4
95.4−1.1 98.3+1.8
90.4−2.3 93.5+0.7
97.6−0.6 98.8+0.6
68.0+3.2 71.9+7.1
86.0+2.2 87.0+3.3
82.8+1.0 86.1+4.2
High
StatA MOON
82.8+0.6 91.5+9.2
77.3+1.9 86.6+11.2
64.1+3.9 64.0+3.8
60.8+3.7 60.1+3.0
95.5+0.8 96.6+2.0
94.8+0.1 97.1+2.4
95.0−1.5 97.2+0.7
90.5−2.2 93.5+0.7
97.8−0.4 98.7+0.5
68.6+3.8 66.6+1.7
85.6+1.9 85.3+1.6
83.0+1.1 85.2+3.3
Very High
StatA MOON
82.8+0.5 91.0+8.8
77.6+2.2 85.2+9.8
60.4+0.2 58.8−1.4
60.8+3.7 60.1+3.0
95.4+0.7 96.2+1.5
94.8+0.1 95.6+0.9
94.9−1.6 97.0+0.5
90.4−2.3 93.4+0.6
97.7−0.5 97.6−0.6
69.0+4.1 65.1+0.3
84.5+0.7 82.7−1.1
82.6+0.7 83.9+2.0
01
02
70.1
91.2
91.0
94.2
75.0
97.3
60.6
78.0
76.5
U
TD D
Ca
Fl
ts
St an
oS
A
Eu r
irc
N
lte
Pe
33.6
ow
Fo od 1
CF 10
1
ch 1
er s1
01
Ca fo
rd
AT
ft
72.9
SU
Method EVA-CLIP
ra
78.0
Im ag
Keff
39
eN
7
et
rs
Table 18. Results on EVA-CLIP (1.1B), batch adaptation with batch size of 1,000. Subscript green indicates improvement, red indicates decline, and gray indicates no change compared with zero-shot performance.
Avg.
Medium
StatA MOON
77.8−0.2 81.1+3.1
71.5−1.5 75.6+2.7
36.9+3.3 36.5+2.8
72.7+2.6 72.7+2.5
91.8+0.6 93.1+1.8
90.8−0.2 93.2+2.2
93.9−0.3 94.8+0.6
77.8+2.7 76.4+1.4
97.1−0.2 97.7+0.4
62.4+1.8 64.8+4.2
79.7+1.7 80.4+2.5
77.5+1.0 78.7+2.2
High
StatA MOON
78.3+0.4 81.3+3.3
73.0+0.1 76.2+3.3
36.2+2.6 35.0+1.4
73.2+3.1 72.7+2.6
92.2+1.0 93.1+1.9
91.6+0.6 92.7+1.7
94.2±0.0 94.2±0.0
78.4+3.4 76.7+1.7
97.4+0.1 97.8+0.5
64.8+4.2 63.5+2.9
80.3+2.4 80.1+2.1
78.2+1.6 78.5+2.0
Very High
StatA MOON
78.7+0.8 81.2+3.2
73.8+0.8 76.1+3.2
35.4+1.8 33.6−0.1
73.2+3.1 72.7+2.6
92.3+1.0 92.9+1.6
91.6+0.6 91.8+0.8
94.2±0.0 94.1−0.1
77.7+2.7 76.0+0.9
97.4+0.1 97.5+0.2
65.5+5.0 63.1+2.5
80.7+2.7 79.5+1.5
78.2+1.7 78.0+1.5
All
1 10 CF U
TD D
Ca
lte
ch 1
10 ow Fl
Pe ts
er s
01 d1 Fo o
01
2
Ca rs or d nf St a
SA T ro
irc
Eu
ra ft
7 N
39 A
Method
SU
Keff
Im
ag
eN et
Table 19. Results on full dataset with all classes. Subscript green indicates improvement, red indicates decline, and gray indicates no change compared with zero-shot performance.
Avg.
CLIP
66.6
62.5
24.7
48.3
65.6
85.9
89.1
70.7
93.2
43.5
67.5
65.2
StatA MOON
69.9+3.3 68.7+2.1
68.7+6.2 65.4+2.9
24.7±0.0 24.4−0.3
67.3+19.0 59.1+10.8
68.0+2.4 67.2+1.6
87.1+1.2 86.7+0.8
92.4+3.3 90.2+1.1
75.2+4.5 72.8+2.1
94.2+1.0 93.4+0.2
48.4+4.9 45.0+1.5
73.5+6.0 70.9+3.4
69.9+4.7 67.6+2.4
31
Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction
Table 20. Results on random class scenarios, where Keff is randomly sampled within [1, min{N , K}]. Subscript green indicates improvement, red indicates decline, and gray indicates no change compared with zero-shot performance.
65.6
85.9
89.1
70.7
93.2
43.5
67.5
65.2
StatA MOON
68.7+2.1 75.0+8.4
64.8+2.3 69.0+6.5
24.1−0.6 22.5−2.2
50.6+2.3 47.8−0.5
68.9+3.3 69.6+4.0
87.1+1.2 89.2+3.3
90.8+1.7 90.0+0.9
72.1+1.4 71.7+1.0
93.8+0.6 93.5+0.3
46.6+3.1 42.5−1.0
68.7+1.2 68.8+1.3
66.9+1.7 67.2+2.0
TD
CF 10
U
D
Fl
St
Eu ro
A
irc ra
ft
7 SU
N
39
eN Im ag
CF 10 1
48.3
TD
Ca lte ch 10 1
10 2 er s
24.7
ow
Pe ts
62.5
an fo rd
66.6
Method
SA T
CLIP
et
Fo od 10 1
Ca rs
(a) Batch Size: 64.
Avg.
U
D
1
01 Ca lte ch 1
10 2 Fl ow er s
Pe ts
Fo o
d1 01
fo rd C St an
SA T Eu
A
ro
t irc
ra f
39 SU
Im
Method
N
ag
eN
7
et
ar s
(b) Batch Size: 128.
Avg.
CLIP
66.6
62.5
24.7
48.3
65.6
85.9
89.1
70.7
93.2
43.5
67.5
65.2
StatA MOON
68.6+2.0 74.1+7.5
64.3+1.8 67.3+4.8
23.6−1.1 22.0−2.7
53.3+5.0 50.6+2.3
67.5+1.9 66.4+0.8
86.9+1.0 87.9+2.0
91.1+2.0 90.7+1.6
71.5+0.8 70.2−0.5
93.8+0.6 93.2±0.0
46.9+3.4 44.1+0.6
67.9+0.4 66.4−1.1
66.8+1.6 66.6+1.4
lte c
D TD
U CF
65.6
85.9
89.1
70.7
93.2
43.5
67.5
65.2
StatA MOON
68.2+1.6 72.9+6.3
64.1+1.6 64.7+2.2
24.1−0.6 23.3−1.4
55.3+7.0 51.9+3.6
66.3+0.7 64.8−0.8
87.4+1.5 89.0+3.1
91.9+2.8 91.7+2.6
73.2+2.5 72.2+1.5
93.8+0.6 93.5+0.3
47.4+3.9 46.5+3.0
68.7+1.2 67.9+0.4
67.3+2.1 67.1+1.9
A
Eu
10
ra ft irc
N SU
Im
1
Fl ow er
48.3
Ca
ts Pe
h1
01
s1 02
Fo od 10 1
Ca rd
24.7
St an
62.5
ro
66.6
Method
39 7
CLIP
ag eN
fo
SA T
et
rs
(c) Batch Size: 256.
Avg.
1 CF 10
TD
48.3
65.6
85.9
89.1
70.7
93.2
43.5
67.5
65.2
StatA MOON
68.1+1.5 70.9+4.3
64.2+1.7 62.4−0.1
24.9+0.2 24.7±0.0
54.5+6.2 52.7+4.4
67.2+1.6 66.5+0.9
87.5+1.6 89.8+3.9
92.5+3.4 91.8+2.7
74.2+3.5 73.1+2.4
93.5+0.3 93.7+0.5
47.1+3.6 47.6+4.1
69.4+1.9 69.1+1.6
67.6+2.3 67.5+2.3
D
Ca l
Eu
irc A
SU
Im
U
24.7
Fl ow
Pe ts
62.5
St an
66.6
Method
ro
CLIP
N
Fo od
te ch 1
01
2 er s1 0
10
1
Ca rd fo
SA T
ft ra
39
ag
eN
7
et
rs
(d) Batch Size: 500.
Avg.
Pe
62.5
24.7
48.3
65.6
85.9
89.1
70.7
93.2
43.5
67.5
65.2
StatA MOON
67.5+0.9 67.9+1.3
65.2+2.7 64.0+1.5
25.4+0.7 25.4+0.7
55.0+6.7 54.7+6.4
68.0+2.4 68.1+2.5
87.2+1.3 89.7+3.8
93.0+3.9 92.0+2.9
75.3+4.6 74.0+3.3
93.3+0.1 94.1+0.9
47.8+4.3 48.1+4.6
70.7+3.2 70.7+3.2
68.0+2.8 68.1+2.8
TD
CF
10 CF
D
U
TD
1
h1 lte c Ca
Fl ow
er
1
ts
Eu ro
A
irc
ra
ft
7 SU
N 39
eN ag Im
01
Fo od 10
66.6
Method
s1
St an fo rd Ca
CLIP
et
SA T
02
rs
(e) Batch Size: 1,000.
Avg.
01
02
10 1 U
D
Ca
lte c
h1
s1 er Fl ow
ts Pe
d1 01 Fo o
Ca St an fo rd
SA T Eu
ro
ft ra irc A
N SU
Method
Im
ag
eN
39 7
et
rs
(f) Batch Size: 2000.
Avg.
CLIP
66.6
62.5
24.7
48.3
65.6
85.9
89.1
70.7
93.2
43.5
67.5
65.2
StatA MOON
68.1+1.5 68.8+2.2
66.1+3.6 66.2+3.7
26.3+1.6 26.3+1.6
56.7+8.4 55.0+6.7
69.6+4.0 69.9+4.3
86.7+0.8 89.4+3.5
93.0+3.9 92.0+2.9
77.0+6.3 75.3+4.6
93.3+0.1 94.2+1.0
47.4+3.9 47.7+4.2
71.5+4.0 71.4+3.9
68.7+3.5 68.7+3.5
32
Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction ImageNet
SUN397
DTD
10.0
Accuracy change vs. m=3
Accuracy gain over StatA
ImageNet (81.6)
7.5 5.0 T=3
2.5 0.0
StatA
−2.5 −5.0 CLIP
1
3
5
7
10
SUN397 (75.5)
DTD (49.0)
1.0 0.5
m=3
0.0 −0.5 −1.0 −1.5 −2.0 −2.5 0
1
Number of iterations T
3
5
10
Number of neighbors m
Figure 4. Hyperparameter sensitivity analysis. Results are reported on batch adaptation, Medium scenario, batch size of 1,000.
Bs=64
Bs=1000
Figure 5. Detailed ablation study on components, over various batch sizes and scenarios. Each reported performance is averaged over all datasets and runs.
J. Additional Analyses Hyperparameter sensitivity. We analyze the sensitivity of MOON to the existing hyperparameters, including iteration number T and the number of neighbors in Laplacian term m in Fig. 4. The results show that MOON is robust to both hyperparameters. Moreover, the performance improves rapidly within the first few iterations and outperforms StatA nearly saturates after T = 3, suggesting fast practical convergence. Together with the dynamic shrinkage mechanism, MOON thus requires no task-specific hyperparameter tuning. We set T = 10 and m = 3 by default for all experiments. Fine-grained ablation analysis. We present a fine-grained ablation study across batch sizes and sparsity levels in Fig. 5. Consistent with our design motivation, the impact of the class-level weight α diminishes as the effective class set becomes denser (e.g., in the Very High scenario). This confirms that α functions precisely as intended: effectively suppressing outlier classes in sparse settings while relaxing constraints when the distribution approaches uniformity. Conversely, the instance-level weight γ shows marginal influence at small batch sizes due to high statistical variance but becomes increasingly significant at larger batch sizes, where it can leverage stable batch statistics to filter unreliable samples effectively. Furthermore, the performance gain from iterative parameter updates tends to saturate in scenarios with large batches and dense classes, suggesting that abundant data naturally provides sufficient empirical evidence for reliable estimation. Overall, 33
Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction
the full MOON framework consistently yields optimal performance. Table 21. Implementation of shrinkage strength βk . (a) Batch adaptation, with batch size of 64. Scenario MOON w/ soft βk MOON w/ hard βk
Very Low
Low
Medium
Avg.
75.3 74.6
73.3 73.1
68.6 69.4
72.4 72.4
(b) Batch adaptation, with batch size of 1,000. Scenario
Medium
High
Very High
Avg.
72.1 72.9
70.3 71.3
68.1 69.2
70.2 71.1
MOON w/ soft βk MOON w/ hard βk
(c) Online adaptation, with batch size of 128. Scenario
Low
Medium
High
Separate
Avg.
MOON w/ soft βk MOON w/ hard βk
64.9 66.5
71.8 71.8
73.6 73.4
74.2 73.9
71.1 71.4
Implementation of βk . We investigate the implementation strategy for the shrinkage strength βk by comparing the standard soft assignment against the hard assignment (i.e., discretization via argmax) on the probability simplex. As shown in Tab. 21, employing hard assignments consistently yields superior robustness and stability. This advantage stems from the inherent property of the softmax operation, which produces non-zero residual probabilities for all classes. In a soft assignment regime, these residuals can accumulate to form misleading counts for absent classes, thereby weakening the necessary shrinkage. By adopting hard assignments, we effectively eliminate this background noise, ensuring that βk accurately reflects the true class presence and enforces strict anchoring for outlier categories.
K. Limitations and Future Work While our MOON demonstrates robust performance and efficiency, there remain promising avenues for future exploration. First, vMF distributions inherently assume isotropy on the hypersphere. Explicitly modeling the anisotropy of VLM representations, for instance, by exploring Fisher-Bingham distributions or other non-isotropic spherical models, could potentially capture more complex feature geometries. Second, MOON can be more deeply integrated with memory banks or caches, enabling more efficient and effective adaptation in sample-wise, online-TTA mode. Finally, the construction of the affinity graph still entails a quadratic complexity with respect to the batch size. Incorporating approximate nearest neighbor search strategies could be beneficial, especially for scaling to large-scale offline adaptation tasks.
34