Gaussian Mean Field Variational Inference can Overestimate Predictive Variance James Odgers 1 2 3 Ben Riegler 4 2 3 Siddharth Swaroop 5 Vincent Fortuin 1 2 3
3
Mean Field Variational Inference (MFVI) is widely understood to underestimate posterior variance. By analysing conjugate Bayesian Linear Regression (BLR), we show that this characterization is incomplete: while MFVI underestimates the variance in parameter space, it can overestimate the predictive variance compared to the exact posterior. We show that if the MFVI posterior underestimates predictive variances in some directions, it necessarily overestimates them in others. Crucially, this overestimation occurs in directions where the training data concentrates. This leads to the surprising result that, for a test point drawn from the training distribution, MFVI’s expected predictive variance exceeds that of the exact posterior. We demonstrate a pathological case of this effect, where the MFVI posterior fails to reduce predictive variance compared to the prior on in distribution data. We connect these results to the Cold Posterior Effect, arguing that varying the temperature can correct this overestimation, yielding predictions closer to those of the exact posterior. We validate our theory on synthetic and real-world regression tasks.
Exact MFVI
2
2 1
θ2
1
x2
arXiv:2606.25745v1 [stat.ML] 24 Jun 2026
Abstract
0
0
−1
−1
−2 −2
0
x1
(a) Input space
2
0
θ1
2
(b) Parameter space
Figure 1. Although the MFVI posterior underestimates variance on average, it overestimates variance along the direction in which the data lie. Here we show this on a simple 2D linear regression, where the input data is restricted to lie on the subspace x1 = x2 (Figure 1a). We see that, for most of the input space, MFVI predictive uncertainty is less than the exact posterior predictive (blue region). At the same time, along the data direction, the opposite is true (red region). This is also the case when looking at the weight-space posterior (Figure 1b), with the exact posterior clearly taking up more volume, but also with the MFVI posterior having a larger variance along the crucial θ1 = θ2 subspace.
which minimises the reverse KL divergence to the true posterior, q ∗ (θ) = arg minq∈Q DKL (q(θ)||p(θ|D)). Practitioners often use Mean Field Variational Inference (MFVI), in which the QPvariational posterior factorises across parameters, q(θ) = p=1 qp (θp ). In practice, the marginal distributions are often restricted to be Gaussian, so qp (θp ) = N (µp , σp2 ). We study this popular setting in this paper.
1. Introduction Bayesian approaches to machine learning offer compelling advantages, including a principled approach to uncertainty estimation, and learning hyperparameters without overfitting (MacKay, 1992). Unfortunately, the full Bayesian posterior, p(θ|D), is usually intractable and needs to be approximated. One of the most popular approaches to approximate this posterior is Variational Inference (VI) (Jordan et al., 1999; Blei et al., 2017). Here, a family of distributions is proposed, Q, from which the distribution q ∗ (θ) is optimized,
MFVI is typically considered to underestimate the uncertainty of the posterior (Minka, 2005; Turner and Sahani, 2011; Blei et al., 2017). This is because it is a mode-fitting approximation in parameter space: the approximate posterior avoids assigning probability mass to regions with low true probability, thereby underestimating the variance in parameter space (Margossian and Saul, 2023). However, in this paper, we argue that this picture is incomplete once the goal is prediction rather than parameter inference. In particular, the predictions from an MFVI posterior can overestimate uncertainty on In-Distribution (ID) data.
1 University of Technology Nuremberg (UTN) 2 Helmholtz AI Munich Center for Machine Learning (MCML) 4 Technische Universität München (TUM) 5 University College London (UCL). Correspondence to: James Odgers <[email protected]>. 3
Illustrative example (Figure 1). Figure 1a shows some simple training data restricted to the subspace x1 = x2 , and Figure 1b shows the exact posterior and MFVI posterior (in parameter space) for this linear regression example. The
rd
Proceedings of the 43 International Conference on Machine Learning, Seoul, South Korea. PMLR 306, 2026. Copyright 2026 by the author(s).
1
Gaussian MFVI can overestimate predictive variance
highly uneven, non-axis-aligned covariance of the training inputs produces a posterior with a strong covariance between θ1 and θ2 in the parameter space. We can see how MFVI appears to underestimate the variance in general: its mass is over a smaller region of the parameter space than the mass of the exact posterior. However, in this example, we are not interested in the parameters learned by the model; instead, we care about the predictions made on test data. A realistic assumption is often that future data points are similar to training data points, such that x1 = x2 . If this is the case, the only direction in parameter space relevant for predictions is θ1 = θ2 , as illustrated by the dashed line in Figure 1b. In this direction, the MFVI posterior overestimates the variance compared to the true posterior. Our main result, given in Theorem 3.7, is to show that the overestimation of predictive variance in MFVI in Bayesian Linear Regression (BLR) is very general.
2. Related Work Variational Inference (VI). VI research has mostly focused on the approximation quality in parameter space, not the approximation quality on predictions (see Blei et al. (2017) for an example of this). One line of related research is Functional VI (Sun et al., 2019; Burt et al., 2021; Cinquin and Bamler, 2025), to allow more interpretable priors, such as GPs, to be applied to BNNs. In our linear regression setting, there is a one-to-one mapping between functions and parameters, so there is no difference in the functional and parametric variational objectives (see Proposition 3, Burt et al., 2021). A functional approach to VI is also common in Gaussian Processes (Titsias, 2009) to improve computational efficiency. We do not consider these approaches in this paper, as these approaches are equivalent to different variational families in parameter space which we do not study here. The paper closest to the theoretical analysis we provide is Margossian and Saul (2023), who studied the theoretical properties of the Gaussian MFVI posterior when the exact posterior is a Multivariate Gaussian. Crucially, this current work focuses on the properties of the predictions of VI, whereas Margossian and Saul (2023) focus on the properties in parameter space.
A Cold Posterior Effect without model mismatch. The Cold Posterior Effect (CPE) from Bayesian Deep Learning (BDL) (Wenzel et al., 2020), describes an effect where artificially reducing the uncertainty of an approximate posterior improves predictive performance. In this work, we add another justification for why we may expect the CPE: lowering the temperature corrects for overestimated predictive variance from MFVI on in-distribution data, producing predictions which match the exact posterior predictions more closely. We demonstrate that the reverse effect is true for Out-Of-Distribution (OOD) data, where a warm posterior matches the exact posterior predictive distribution more closely and performs better on predictive tasks. Contributions.
Cold Posterior Effect (CPE). There are several works proposing explanations for the CPE, showing how this effect can naturally arise from data augmentation (Izmailov et al., 2021; Nabarro et al., 2022; Bachmann et al., 2022), misspecified likelihoods (Adlam et al., 2020; Aitchison, 2021), misspecified priors (Fortuin et al., 2022; Kapoor et al., 2022; Marek et al., 2024), or a combination of effects (Zeno et al., 2020; Noci et al., 2021). We do not disagree with these causes; our paper provides an additional novel explanation, in which the model has a well-specified likelihood and prior, and no data augmentation is used. Instead, we propose that the CPE allows the MFVI approximation to more closely match the exact posterior predictions for certain tasks. Zhang et al. (2024) argue that the CPE is correcting for underconfident predictions, which we agree with and extend by providing a mechanistic understanding of this effect with MFVI. Many MFVI algorithms in deep learning temper their likelihood function, but do not provide any theoretical analysis or reasoning (e.g., Osawa et al., 2019; Ashman et al., 2022). We became aware of contemporaneous work by Harvey et al. (2025) showing that the standard MFVI-ELBO objective produces underconfident predictions, though their focus is on hyper-parameter learning, which we hold fixed.
In this paper, our main contributions are:
• We provide a theoretical analysis of conjugate Bayesian Linear regression, showing that the MFVI posterior overestimates the exact posterior variance along the first principal component of the training data, and that the average predicted variance at the training data points is higher for MFVI than the exact posterior. • We theoretically and empirically examine the predictive performance of MFVI on a pathological setup and demonstrate that in the high-dimensional limit, the MFVI posterior can fail to reduce the predictive variance at all compared to the prior. • We show theoretically and empirically that both a Cold Posterior Effect and Warm Posterior Effect can occur in linear regression models approximated with MFVI, and argue that both of these effects can be understood as scaling the predictive variance of the MFVI approximation to better match the exact posterior predictions.
GenBayes. Generalized Bayes centres around the idea that traditional Bayesian inference methods may not give models the best predictive properties; a tempering strategy (Aitchison, 2021), in which the likelihood and the prior regularisation terms of the VI objective are weighted differ2
Gaussian MFVI can overestimate predictive variance
ently, can produce better predictions (Pitas and Arbel, 2022; McLatchie et al., 2025). Our work is distinct from tempered objectives. Instead, we consider an objective approximating an exact cold posterior, which cannot be written with a tempered objective, as discussed in detail in Section 4.
This objective is minimised by the true posterior; however, this is commonly not available for computational reasons. Instead, the variational posterior is constrained to factorise over its P dimensions. In the Multivariate Gaussian case, this requires constraining the variational posterior to be diagonal, which is equivalent to fixing the eigenbasis to be axis-aligned. This allows us to write
3. MFVI Overestimates Predictive Uncertainty for the Empirical Distribution Problem Statement. Assume we have a model where inputs x ∈ RP and outputs y ∈ R are related by the linear predictor θ ∈ RP , such that ⊤
y = θ x + ϵ,
2
ϵ ∈ R ∼ N (0, σ ).
S=
Σ=
(5)
where ep are the canonical basis vectors which contain a single 1 at position p and are zero otherwise.
(1)
In this form, the goal of fitting the variational posterior is given by finding the optimal parameters of m∗ and {d∗p }P p=1 . The optimal parameters for these, formalised in the following two lemmas, can be found as functions of the posterior mean, covariance eigenvalues, and the inner product of the orthonormal bases of the posteriors, vqT ep . The proofs for these can be found in Appendix A.1. Lemma 3.1 (Optimal variational mean). For any positive definite posterior covariance, the optimal mean is the mean of the true posterior:
True Posterior. In this work, we are interested in comparing to the true posterior
dp ep e⊤ p,
p=1
We assume that we have a training dataset of the form N ×P D = {xn , yn }N , Y ∈ RN } drawn n=1 = {X ∈ R from this distribution. We will call thePempirically obN served distribution of inputs p̂(x) = N1 n=1 δ(x − xn ), and assume that the training data was mean-centred, so Ep̂(x) [x] = 0. Throughout this paper, we will assume that the prior, θ ∼ N (0, α−1 I), and likelihood, σ 2 , are well specified and known to us.
p(θ|D) = N (µ, Σ),
P X
m∗ = µ.
(2)
(6)
Lemma 3.2 (Optimal variational posterior eigenvalues are harmonic means). The optimal variational covariance, S ∗ , is given by the condition
−1 1 ⊤ 1 ⊤ X X + αI , µ = Σ X Y . σ2 σ2
P X wpq 1 ⊤ −1 = e Σ e = p p d∗p δq q=1
Specifically, we are interested in comparing the predictions for the output, y, for a given test point, x, from the true posterior and approximate posteriors.
where wpq = (vqT ep )2 and {1, . . . , P }.
Diagonalised True Posterior Covariance. We will work with the diagonalised covariance matrix. For this, we define Σ = V ∆V ⊤ , where ∆ = diag(δ1 , . . . , δP ), δp ∈ R+ , is a diagonal matrix and the columns of V = [v1 , . . . , vP ], vp ∈ RP , form a complete, orthonormal basis. We will use the expression P X Σ= δq vq vq⊤ . (3)
∀ p ∈ {1, . . . P },
PP
q=1 wpq
(7)
= 1 ∀ p ∈
While the values of the optimal variational posterior are well known in the literature (Turner and Sahani, 2011; Margossian and Saul, 2023), we are unaware of prior work describing the variational posterior eigenvalues as a weighted harmonic mean of the eigenvalues of the exact posterior. Later, this will be important for our theoretical arguments.
q=1
For a test input location, x, we are interested in comparing the predictive distributions excluding aleatoric uncertainty, which are given by
The variational posterior. We are interested in finding an approximate posterior q(θ) = N (m, S) by minimising the reverse Kullback-Leibler (KL) divergence to the true posterior. For our linear regression model, the true posterior is Gaussian, so the objective to be minimised is
p(f |D) = N (µ⊤ x, x⊤ Σx),
q(f ) = N (m∗⊤ x, x⊤ S ∗ x)
(8)
for the exact and MFVI posteriors, respectively. As the optimal variational and exact posterior means are identical, the only difference in these distributions is in the variance of the predictions, which is where we focus from now on.
1 DKL (q(θ)||p(θ|D)) = (µ − m)⊤ Σ−1 (µ − m) 2 + T r(Σ−1 S) − log det S + log det Σ − P . (4) 3
Gaussian MFVI can overestimate predictive variance
This direction is the span of the eigenvector of the minimum eigenvalue of the true posterior, that is, for a test point x̃ ∈ span(vq∗ ) where δmin = vq⊤∗ Σvq∗ . Lemma 3.4 (Overestimation of predictive variance in one direction of the input space). For points in the input space defined by x̃ ∈ span(vq∗ ), the predictive variance for the exact posterior and the MFVI posterior are governed by the inequality x̃⊤ S ∗ x̃ ≥ x̃⊤ Σx̃. (11)
3.1. Theoretical Analysis of Predictions from MFVI This section proves theoretical results about the relationship between predictions from the MFVI and the exact posterior. We start by confirming the traditional story that MFVI underestimates variance, i.e. is overconfident, showing this is the case when the test input is isotropic. After this, we focus on the ways in which this traditional story is misleading. First, we show that there must be at least one direction in which the MFVI overestimates variance and, for spherical priors, this is the direction of maximum variance of the training data. After this, we show that the degree of overand underestimation of the predictive variance is coupled together. Finally, we give our main result: MFVI overestimates variance, i.e. is underconfident, for test points similar to the training data. Formal proofs of all our theorems are in Appendix A.
The intuition for this is that the exact posterior predictive variance will be δmin , which, because the eigenvalues of the MFVI posterior are a harmonic mean of the exact eigenvalues, must be lower than the lowest eigenvalue and the lowest predictive variance of the MFVI posterior. In the common case of spherical priors, where the prior variance is given by α−1 I, the relative size of the posterior eigenvalues are entirely determined by the variance of the training data. This means that the direction in which this underestimation occurs is of great significance: it is the first principal component of the training data. More formally we can state the following theorem. Theorem 3.5. Consider the conjugate Bayesian linear model of Equation (1) with a spherical prior θ ∼ N (0, α−1 I), and let w denote the first principal component of the training inputs X. Then for any test point x̃ ∈ span(w), the predictive variances of the MFVI and exact posteriors satisfy
MFVI predictions underestimate uncertainty for isotropic test points. We begin by considering the setting where the test distribution is isotropic, x ∼ N (0, IP ). This corresponds to the assumption that test points are equally likely to arrive from any direction in the input space, a setting in which we would expect MFVI’s tendency to underestimate variance to manifest clearly. Indeed, in this setting, we recover the standard result that MFVI is overconfident. To see this, we note that the expected predictive variance under an isotropic test distribution is simply the trace of the posterior covariance: Ex∼N (0,IP ) x⊤ Ax = Tr (A)
x̃⊤ S ∗ x̃ ≥ x̃⊤ Σx̃.
(9)
The proof follows from noting that, for spherical priors, the exact posterior shares an eigenbasis with the gram matrix of the training data, X ⊤ X, so w = vq∗ and Lemma 3.4 can be applied to give the desired result.
for any matrix A. The following lemma then follows directly from proving that the trace of the exact posterior is greater than the MFVI posterior, which we do in Lemma A.3. Lemma 3.3. For a test point x ∼ N (0, IP ), the predictive variances of the MFVI posterior and exact posterior are governed by Ex∼N (0,IP ) x⊤ Σx ≥ Ex∼N (0,IP ) x⊤ S ∗ x .
(12)
MFVI under- and overestimates uncertainties similarly. We can take the above result and strengthen it by showing that the degree of overestimation and underestimation of the predictive density are linked. To do this, we introduce the ratio of the predictive variances at test point x,
(10)
x⊤ S ∗ x . (13) x⊤ Σx R(x) > 1 implies an overestimated uncertainty and R(x) ≤ 1 implies underestimated uncertainty. We can now consider the average value of this new quantity across the complete eigenbasis of the exact posterior, and we will find the following surprising result. Lemma 3.6 (Calibrated MFVI). Consider the set of eigenvectors of the true posterior {vq }P q=1 . For these points
This result aligns with the standard understanding that MFVI underestimates uncertainty. However, the assumption of isotropic test data is often unrealistic: in practice, we expect test points to follow a similar distribution to the training data. We now turn to this more realistic setting.
R(x) =
MFVI overestimates uncertainty along the first principal component of the training data. The first result we use to argue that MFVI overestimates uncertainty is to show that there is a direction in the input space where, if MFVI and exact posterior predictions are not equal, the MFVI posterior overestimates the predictive uncertainty.
P
1 X R(vq ) = 1. P q=1 4
(14)
Gaussian MFVI can overestimate predictive variance
While we are unable to provide an intuitive explanation for why this is the case, the result can be found by first showing that Tr Σ−1 S ∗ = P , as we do in Lemma A.4, and then performing straightforward algebra.
the form N (0, α−1 IP ) and the likelihood has the form N (y|θ⊤ x, σ 2 ), the optimal parameters are
This constraint, that the average value of R(x) is constant across eigenvectors of the true posterior, can cause problems for predicting in-distribution data. In particular, in high-dimensional settings, where many eigenvector directions have underestimated variance, the degree of overestimation in other directions must be large to compensate. We demonstrate an extreme version of this in Section 5.1.
where m∗ and S ∗ are the solutions at T = 1 given in Equations (6) and (7). We prove this in Appendix A.6.
qT∗ (θ) = N (m∗ , T S ∗ ),
Different Temperatures for Different Tasks. When making predictions, there are several possible settings which would lead to different optimal temperatures. Intuitively, given the overestimation of variance in Theorem 3.7, we may expect that there is an optimal temperature T < 1 which will make the T -MFVI and exact posterior predictive distributions match most closely on in-distribution data. However, by Lemma 3.6, if there are some directions in the input space where the predictive variance is overestimated, there must also be locations where the predictive variance is underestimated. We can therefore deduce that there may be a temperature T > 1 which would make out-of-distribution predictions better calibrated.
MFVI overestimates predictive variance for data with the empirical covariance. So far, we have shown the following results: from Lemma 3.3 we can expect that there are a lot of directions in the input where the variance is underestimated, from Lemma 3.4 we can expect that for spherical priors the highest variance of the training data is underconfident, and from Lemma 3.6 we know that the degrees of this overestimation and underestimation are coupled in some way. These results provide some intuition for our main theorem, which says that, for BLR problems with a spherical prior, MFVI overestimates predictive uncertainty for test data points distributed according to the empirical distribution of the training data, p̂(x). Theorem 3.7. For test data points distributed according to the empirical distribution x ∼ p̂(x), the difference in the expected predicted variance of the MFVI and exact posterior is given by (15) Ex∼p̂(x) x⊤ Σx − x⊤ S ∗ x ≤ 0.
Furthermore, it is not immediately obvious with which measure to compare the variational and exact posteriors. There are multiple ways to measure distances between predictive distributions, each of which captures meaningful but different notions of similarity. In our experiments in Section 5.2, we examine both ID and OOD data, and consider multiple notions of distances to the true posterior. We confirm that we should expect cold temperatures (T < 1) to perform well for ID data, and warmer temperatures for OOD data, and that this is generally true across multiple measures of discrepancy between distributions.
In our opinion, this is a pretty remarkable result: while MFVI can in most senses be considered to underestimate the epistemic uncertainty, in the very important case of predicting on test data which shares covariance with the training data, MFVI overestimates uncertainty.
Cold Posteriors vs. Tempered Posteriors. There are two distinct types of sharpened posteriors: tempered posteriors, in which the prior is kept constant but the likelihood is scaled, and cold posteriors, in which both the likelihood and the prior terms are scaled (Aitchison, 2021). VI methods typically favour using a tempered approach, rather than the approach taken here, which fits naturally within the ELBO objective by simply scaling the expected log-likelihood compared to the KL term. It is not generally possible to write the cold posterior objective in such an elegant way, albeit with an exception for the Gaussian priors used here (Aitchison, 2021). In general, if the cause of the CPE is one of those in Section 2, we would advocate for the use of a tempered posterior. However, this paper is interested in how well predictions from an approximate posterior match predictions from an exact posterior, and in this case, we believe that there is a good reason to prefer the cold posterior to the tempered posterior.
4. Cold MFVI posterior predictions can match Bayesian predictions The Cold Posterior Objective for MFVI. Cold posteriors describe exponentiated posteriors, pT (θ|D) ∝ p(D|θ)1/T p(θ)1/T , where T < 1. This exponentiation corresponds to sharpening the exact posterior, so more of the mass is concentrated around the Maximum A Posteriori (MAP) estimate. If, rather than minimising the KL divergence to the exact posterior, we minimise the KL divergence to the cold posterior, we get a variational posterior qT∗ (θ) = min DKL (q(θ)||pT (θ|D)) . q(θ)∈Q
(17)
(16)
Concretely, the cold posterior MFVI shares a mean with the T = 1 posterior in our case (Multivariate Gaussians), while the tempered posterior does not. If we applied the tempered
We refer to this solution as the T -MFVI posterior. For the specific conjugate case where the prior has 5
Gaussian MFVI can overestimate predictive variance Prior Pred
Exact Post (T = 1)
MAP (Noise)
MFVI (T = 1)
0.75
−2
0.50
−3
0.25
−4
0.00
µ
Test NLL
Predictive Density
−1
1.00
Log10(T )
0
1.25
>x
MFVI NLL 95% CI
>x
σ σ −6 >x−3 µ
µ
>x
µ
σ σ +3 >x+6 µ
>x
µ
σ σ −6 >x−3 µ
(a) P = 2
>x
µ
>x
µ
Test NLL
0.6 0.29 0.4
10−4
10−3
10−2
10−1
Temperature T (a) P = 2
100
10−4
10−3
10−2
4
2
2 101
102
100
101
102
Temperature T (b) P = 1024
posterior is informed by the data, while the remaining P − 1 directions retain the prior variance α−1 . Figure 1 showed that MFVI overestimates predictive variance along the data direction x1 = x2 ; we now show that this overestimation becomes increasingly severe as the dimension grows, to the point where MFVI fails to reduce predictive variance compared to the prior at all.
T =1
0.30
0.28
4
Figure 4. Test NLL as a function of temperature for OOD test data orthogonal to the training subspace, averaged over 10,000 repetitions, for (a) P = 2 and (b) P = 1024. For P = 2, the pattern is reversed compared to the ID case: warm posteriors (T > 1) improve performance. For P = 1024, MFVI at T = 1 already closely matches the exact posterior, so the optimal temperature is approximately T ≈ 1.
Figure 2. Posterior predictive densities at an in-distribution test point x = √1P 1P for (a) P = 2 and (b) P = 1024. For P = 2, MFVI at T = 1 (dashed line) closely matches the exact posterior (green), while for P = 1024, MFVI at T = 1 is nearly indistinguishable from the prior (blue), confirming Proposition 5.1. In both cases, an appropriate cold temperature (light-coloured lines) recovers the exact posterior predictive. Exact Post (T = 1) MAP NLL
6
Temperature T (a) P = 2
(b) P = 1024
MFVI NLL 95% CI
6
100
σ σ +3 >x+6 µ
T =1
Exact Post (T = 1) MAP NLL
10−1
Proposition 5.1. In this setting, as P → ∞, for any test input x ∈ RP , the posterior predictive variance of the MFVI posterior will be given by
100
Temperature T (b) P = 1024
Figure 3. Test NLL as a function of temperature for ID test data, averaged over 10,000 repetitions, for (a) P = 2 and (b) P = 1024. Horizontal lines indicate the NLL of the exact posterior and the MAP solution. In both cases, an optimal temperature exists where the T -MFVI predictive matches the exact posterior. In high dimensions, this optimal temperature is much lower, the improvement from tuning T is much larger, and the MFVI solution at T = 1 performs far worse than the MAP.
x⊤ S ∗ x = α−1 ||x||22 ,
(18)
where α is the prior precision. The proof is in Appendix A.7, and the intuition is as follows: Each MFVI variance d∗p is a weighted harmonic mean of the exact posterior eigenvalues (Lemma 3.2). In our rank-1 setting, these eigenvalues consist of one small value δmin (corresponding to the data direction) and P − 1 values equal to the prior variance α−1 . As P grows, the prior eigenvalues dominate the harmonic mean, and the information from the single data direction is diluted across all P axis-aligned components.
posterior, we would face a trade-off between the quality of our posterior predictive and the posterior mean and variance. The cold posterior avoids this trade-off by maintaining the same mean for all values of T .
Figure 2 illustrates this effect for in-distribution test data. We evaluate at a test point x = √1P 1P , which lies in the data subspace and has unit norm. The figure shows posterior predictive densities for P = 2 (left) and P = 1024 (right). We highlight four key distributions: the prior predictive (which has not seen any data), the exact posterior predictive, the MAP prediction (which has zero epistemic uncertainty), and the MFVI posterior predictive at T = 1. For P = 2, the MFVI predictive closely matches the exact posterior: the approximation is working well. However, for P = 1024, the MFVI predictive at T = 1 is nearly indistinguishable from the prior, despite the exact posterior
5. Experiments 5.1. Pathological example: Low Rank Linear Regression We now examine an extreme case that makes the pathology from Section 3 maximally severe.1 Consider training data confined to a non-axis-aligned rank-1 subspace: x = ε1P where ε ∼ N (0, β). This is a direct generalization of the illustrative example from Figure 1 to arbitrary dimension P . The setup ensures that only a single direction of the exact 1 We make our code available at https://github.com/ jamesacodgers/mfvi-cpe.
6
Gaussian MFVI can overestimate predictive variance
remaining well-concentrated. This confirms the limiting behaviour of Proposition 5.1: in high dimensions, MFVI has failed to incorporate the information from the training data into its predictions.
1.5 1.0 0.5
y
The coloured lines show T -MFVI predictives for a range of temperatures. As T → 0, these interpolate smoothly from the MFVI solution toward the MAP prediction. Since the exact posterior predictive lies between these two extremes, there exists a critical temperature T ∗ < 1 for which the T -MFVI and exact posterior predictions match exactly. Crucially, because all training data lies on a single subspace, this temperature simultaneously corrects the predictive variance for all in-distribution test points.
0.0
−0.5 −1.0 −1.5
−1.5
−1.0
−0.5
0.0
x
0.5
1.0
1.5
(a) Low dimensional feature space (Q = 16) 1.5
Figure 3 shows the test negative log-likelihood (NLL) as a function of temperature, averaged over 10,000 repetitions. The test set is generated ID from the same distribution as the training data, so test points also lie in the 1P subspace. For both P = 2 and P = 1024, lowering the temperature initially improves performance until a minimum is reached, where the T -MFVI predictive closely matches the exact posterior. Beyond this point, further cooling degrades performance as the predictive approaches the MAP solution.
1.0
y
0.5 0.0
−0.5 −1.0 −1.5
Three key differences emerge between low and high dimensions. First, the optimal temperature for P = 1024 is much lower than for P = 2, reflecting the greater severity of the MFVI overestimation in high dimensions. Second, the potential improvement from tuning T is far greater for P = 1024: the gap between MFVI at T = 1 and the exact posterior is substantial, whereas for P = 2 it is small. Third, the penalty for setting T too low is much smaller than for setting T too high. In the limit T → 0, the T -MFVI predictive converges to the MAP solution, which has zero epistemic uncertainty, while at T = 1 the MFVI predictive can approach the prior (as shown in Proposition 5.1). For well-specified problems where the data is informative, the exact posterior variance lies much closer to zero than to the prior variance, so the MAP incurs a far smaller penalty than MFVI at T = 1.
−1.5
−1.0
Exact mean Exact CI
−0.5
0.0
x
MFVI (T = 1) Data
0.5
T = 10−3 T = 10−2
1.0
1.5
T = 10−1
(b) High dimensional feature space (Q = 1024) Figure 5. Predictions and credible intervals for fixed basis function regression with (a) Q = 16 and (b) Q = 1024 basis functions. MFVI with T = 1 significantly overestimates the variance of the exact posterior on in-distribution data, and lower temperatures correct this.
to the single data direction, and MFVI averages over all of them, the MFVI predictive variance in any one orthogonal direction is already close to the prior, and hence close to the exact posterior. The optimal temperature is therefore approximately T ≈ 1. These results highlight that the optimal temperature for matching the posterior prediction is task-dependent and, for our setting at least, the CPE arises as a natural method to match the exact Bayesian prediction on in-distribution prediction tasks.
To emphasize that no single temperature is universally optimal, we now consider predictions for test points orthogonal to the training data. Specifically, we evaluate at x = √1P 1± P,
where 1± P is a balanced binary vector with half the entries equal to +1 and half equal to −1. This direction is orthogonal to the training subspace, so the exact posterior does not reduce predictive uncertainty compared to the prior. Figure 4 shows the test NLL for this OOD setting. For P = 2, the pattern from the ID case is reversed: warm posteriors (T > 1) now yield predictions closer to the exact posterior. This is because MFVI underestimates the predictive variance in directions orthogonal to the data, and increasing the temperature corrects this. For P = 1024, the picture is different: since there are so many directions orthogonal
5.2. Basis function regression In order to illustrate that our results from linear regression can provide a useful intuition for general Machine Learning problems, we switch our setting to Basis Function Regression. Here, we assume that we have data D = {X, Y }, but we want to find relationships which would be non-linear in the original space. This can be done by defining a function ϕ : RP → RQ , q = 1, . . . , Q, and modelling the relationship between input and outputs by a weighted sum of these 7
Gaussian MFVI can overestimate predictive variance
functions, so ⊤
yn = θ ϕ(x) + ϵ.
We are interested in establishing which temperature T ∗ most closely matches the exact Bayesian model. However, as discussed in Section 4, it is not clear how to decide how to measure the closeness between these distributions. For this reason we consider a number of statistical divergences between the true and the T -MFVI posterior predictives: forward KL (DKL (p(y|x, D)||qT (y|x))), reverse KL (DKL (qT (y|x)||p(y|x, D))), α divergences (Dα , for α = 0.5), Wasserstein-2 distance (W2 (qT (y|x), p(y|x, D))), and the distance between predictive variances as measured by the Frobenius norm (∥Σp − ΣqT ∥2F ). The temperature T ∗ minimising a given divergence is recorded for 15 independent train, ID test and OOD test splits of the data, and shown in Figure 6.
(19)
This problem is identical to the linear regression one introduced in Equation (1), but with inputs ϕ(x) rather than x. This type of system would typically be fitted with kernel methods, and be described as a Gaussian Process (GP) (Rasmussen and Williams, 2005). Here, we wish to see the effects of MFVI in the parameter space, so we restrict ourselves to finite values of Q, and calculate explicit values for q(θ), just as in the linear regression case. For our experiments, the function ϕ(·) can either be fixed or have hyperparameters which can be learned by maximizing the marginal likelihood, as with any other GP. We use Radial Basis Functions (RBF) as our basis functions for our experiments, with the lengthscale and centroids as hyperparameters. See appendix B for more details.
A clear pattern across all divergences and all datasets can be observed, where for the training and ID test sets, all measures of divergence are minimised by T < 1. Meanwhile, predictions on the OOD test set always require higher temperatures than the ID test set, and, while the precise temperature will depend on the divergence used and distribution of training and OOD test points, often take values of T > 1. This pattern is nicely in line with our main theoretical result, Theorem 3.7, stating that the MFVI prediction will overestimate predictive uncertainty on the training data, but must underestimate predictive variance in directions with less variance in the training data to maintain the calibration criterion we give in Lemma 3.6.
Sinc toy data. To illustrate the underconfidence pathology we fit a sinc(x) function using RBF basis functions with fixed lengthscale 0.25. This lengthscale is quite high compared to the spacing of the data, meaning that the inputs in feature space, ϕ(x), are close together and the empirical input covariance will have very different eigenvalues. We fit the data twice: once with Q = 16 centroids, and once with Q = 1024 and show the results in Figure 5. The true posterior prediction is very similar for both of these, however, the MFVI predictions differ markedly. The MFVI prediction with Q = 16 matches the true posterior well, but the MFVI prediction with Q = 1024 is highly overinflated, matching the intuition from Section 5.1 that high-dimensional settings amplify the effect of posterior prediction mismatch.
Appendix B contains additional details on the divergences, data pre-processing and hyperparameter selection. 5.3. MFVI underconfidence in BNNs While our paper has focused on the linear setting, the CPE is primarily of interest in Bayesian Deep Learning (BDL). A natural question is therefore whether the results we identify here also hold in BNNs. While a complete investigation of BNNs is beyond the scope of this work, there are some conceptual similarities between the setting we have studied and BDL, which we lay out here.
UCI regression. To show how our theoretical insights translate to real-world problems, we consider several regression tasks from the UCI repository (Kelly et al., 2023), specifically the set used in Foong et al. (2019). Each data set is split into a train set, an in-distribution test set (ID Test) and an out-of-distribution test set (OOD Test). Using Q = 500 RBF basis functions, we train the hyperparameters as described above. We then analytically determine the mean and covariance of both the true and the T -MFVI posteriors (see Equations (2) and (17)) as well as the associated posterior predictive distributions. We distinguish the joint posterior predictive for a test set, (X, Y ), and the marginal one, for a test point, (x, y). The marginal true posterior predictive at an input, x, is given by p(y | x, D) = N (y; µ⊤ x, x⊤ Σx + σ 2 ),
The key insight from our paper is that MFVI will overestimate predictive variance on ID data and underestimate predictive variance on OOD data in the linear regression setting. Both of these effects have previously been observed in MFVI-BNNs for special settings: Foong et al. (2019) showed that MFVI-BNNs’ predictive variance underestimates the true posterior in certain OOD inputs, and Coker et al. (2022) showed that certain MFVI-BNNs’ predictions on ID data revert to the prior as the network width increases.
(20)
Additionally, a key prediction of our theory is that the MFVI over- and underestimation of predictive variance will be worse when the regression takes place in a highdimensional parameter space, but the data concentrates near
with µ and Σ from Equation (2). In addition to the marginal predictions we also consider the distance between the joint predictive distributions, where the covariances are XΣX ⊤ + σ 2 I and T XS ∗ X ⊤ + σ 2 I respectively. 8
Gaussian MFVI can overestimate predictive variance 1.0
0.2
0.5
0.0
0.0
y
log10 (T ∗ )
Marginal divergences
−0.2
−0.5
−0.4
−1.0
−0.6
−6
−4
−2
0
2
4
6
4
6
x
(a) Small BNN (depth 2, width 16)
Joint divergences 1.0 0.5
0.5
y
log10 (T ∗ )
0.0 −0.5
0.0 −0.5
−1.0
−1.0
−1.5
−6
−2.0
DKL
(p |
ID Test
−2
0
2
x
(b) Larger BNN (depth 2, width 512)
2
) ) | p) qT ) kF p, q T p | qT (q T W 2( − Σq T Dα( DKL kΣ p Train
−4
Figure 7. Plot showing the mean and credible interval for a small (Figure 7a), and large (Figure 7b) BNN trained with IVON. Similar to the basis function regression in Figure 5, as the number of parameters of the BNN increases the MFVI model becomes less confident over the region where the data was observed.
OOD Test
Figure 6. The temperature minimising a given divergence between the true posterior predictive, p(·|·, D), and the T -MFVI predictive, qT (·), on the train, ID, and OOD test sets. It can be seen that the optimal temperatures for the train and ID test set are significantly lower than for the OOD test set. For the marginal divergences, the OOD points frequently require a warm posterior (T > 1), while predictions at the train and ID test points benefit from a cold posterior (T < 1). For the joint divergences, this ordering of T ∗ across the three sets is preserved. Results are shown on the UCI kin8nm data set with Q = 500 fixed RBF basis functions.
data described here may explain the success of recently developed non-mean field variational posteriors (Fadel et al., 2025).
6. Conclusion a low-dimensional subspace. It is revealing to consider the loss landscape implied by this structure. The Hessian of the log-posterior will be nearly flat in most directions, controlled only by the prior, and sharp in only the few directions where the data provides information. This is precisely the Hessian structure of the loss that has been observed in deep learning (Sagun et al., 2016).
In our paper, we have compared the predictions from the MFVI posterior and the exact posterior in conjugate linear regression. We have provided both theoretical and empirical evidence that, while MFVI shrinks variance in parameter space and for OOD predictions, MFVI can inflate predictive variance on ID data, which can lead to a strong CPE when data is constrained close to a low-rank linear subspace. While it is very common for approximate inference algorithms to assess predictions on both ID and OOD data (e.g., Shen et al., 2024; Fadel et al., 2025), much of the theoretical analysis of variational inference focuses on the parameter space and ignores the predictive behavior (Wainwright and Jordan, 2008; Margossian and Saul, 2023; 2025; Margossian et al., 2025; Zellinger and Vergari, 2026; Marks et al., 2026). We show that this analysis can miss important effects when the inputs are highly structured, and we hope that our work can influence other researchers to analyze the impacts of approximate Bayesian algorithms on predictions at different input locations.
To provide preliminary evidence that this mechanism operates beyond conjugate models, we train two BNNs on 1D sinusoidal data using IVON (Shen et al., 2024), an MFVI optimizer for deep learning. Figure 7 compares a small network (depth 2, width 16) with a larger network (depth 2, width 512). Despite the means of both networks fitting the data reasonably well, the larger network exhibits substantially wider credible intervals. This mirrors the basis function regression result in Figure 5: increasing the dimensionality of the parameter space while keeping the data fixed leads to a higher predictive variance. We believe that the overestimation of predictive variance on in-distribution 9
Gaussian MFVI can overestimate predictive variance
A. Y. Foong, Y. Li, J. M. Hernández-Lobato, and R. E. Turner. ’in-between’ uncertainty in Bayesian neural networks. In ICML Workshop on Uncertainty and Robustness in Deep Learning, 2019.
Impact Statement This paper presents work whose goal is to advance the field of machine learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
V. Fortuin, A. Garriga-Alonso, S. W. Ober, F. Wenzel, G. Rätsch, R. E. Turner, M. van der Wilk, and L. Aitchison. Bayesian neural network priors revisited. In International Conference on Learning Representations, 2022.
Acknowledgments We thank Thomas Möllenhoff and Emtiyaz Khan for helpful discussions. VF was supported by the Branco Weiss Fellowship.
E. Harvey, M. Petrov, and M. C. Hughes. Learning hyperparameters via a data-emphasized variational objective. arXiv preprint arXiv:2502.01861, 2025.
References B. Adlam, J. Snoek, and S. L. Smith. Cold posteriors and aleatoric uncertainty. In ICML Workshop on Uncertainty and Robustness in Deep Learning, 2020.
P. Izmailov, S. Vikram, M. D. Hoffman, and A. G. G. Wilson. What are Bayesian neural network posteriors really like? In International conference on machine learning, pages 4629–4640. PMLR, 2021.
L. Aitchison. A statistical theory of cold posteriors in deep neural networks. In International Conference on Learning Representations, 2021.
M. I. Jordan, Z. Ghahramani, T. S. Jaakkola, and L. K. Saul. An introduction to variational methods for graphical models. Machine learning, 37(2):183–233, 1999.
M. Ashman, T. D. Bui, C. V. Nguyen, S. Markou, A. Weller, S. Swaroop, and R. E. Turner. Partitioned variational inference: A framework for probabilistic federated learning, 2022.
S. Kapoor, W. J. Maddox, P. Izmailov, and A. G. Wilson. On uncertainty, tempering, and data augmentation in Bayesian classification. Advances in neural information processing systems, 35:18211–18225, 2022.
G. Bachmann, L. Noci, and T. Hofmann. How tempering fixes data augmentation in Bayesian neural networks. In International Conference on Machine Learning, pages 1244–1260. PMLR, 2022.
M. Kelly, R. Longjohn, and K. Nottingham. The UCI machine learning repository. https://archive.ics. uci.edu, 2023. Accessed: 2026-01-28. D. P. Kingma and J. L. Ba. Adam: A method for stochastic gradient descent. In ICLR: international conference on learning representations, pages 1–15, 2015.
D. M. Blei, A. Kucukelbir, and J. D. McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association, 112(518):859–877, 2017.
D. J. MacKay. Bayesian interpolation. Neural computation, 4(3):415–447, 1992.
D. R. Burt, S. W. Ober, A. Garriga-Alonso, and M. van der Wilk. Understanding variational inference in functionspace. In Symposium on Advances in Approximate Bayesian Inference, 2021.
M. Marek, B. Paige, and P. Izmailov. Can a confident prior replace a cold posterior? arXiv preprint arXiv:2403.01272, 2024.
T. Cinquin and R. Bamler. Well-defined function-space variational inference in Bayesian neural networks via regularized KL-divergence. In Uncertainty in Artificial Intelligence, pages 752–776. PMLR, 2025.
C. C. Margossian and L. K. Saul. The shrinkage-delinkage trade-off: An analysis of factorized Gaussian approximations for variational inference. In Uncertainty in Artificial Intelligence, pages 1358–1367. PMLR, 2023.
B. Coker, W. P. Bruinsma, D. R. Burt, W. Pan, and F. DoshiVelez. Wide mean-field Bayesian neural networks ignore the data. In International Conference on Artificial Intelligence and Statistics, pages 5276–5333. PMLR, 2022.
C. C. Margossian and L. K. Saul. Variational inference in location-scale families: Exact recovery of the mean and correlation matrix. In International Conference on Artificial Intelligence and Statistics, pages 3466–3474. PMLR, 2025.
S. G. Fadel, H. Roy, N. Krämer, Y. Zainchkovskyy, S. Syrota, A. V. Mahou, C. H. Ek, and S. Hauberg. VIKING: Deep variational inference with stochastic projections. In Advances in Neural Information Processing Systems, 2025.
C. C. Margossian, L. Pillaud-Vivien, and L. K. Saul. Variational inference for uncertainty quantification: An analysis of trade-offs. Journal of Machine Learning Research, 26(202):1–41, 2025. 10
Gaussian MFVI can overestimate predictive variance
D. Marks, D. Paccagnan, and M. van der Wilk. Symmetry guarantees statistic recovery in variational inference. arXiv preprint arXiv:2604.18310, 2026.
M. J. Wainwright and M. I. Jordan. Graphical models, exponential families, and variational inference. Foundations and Trends® in Machine Learning, 1(1-2):1–305, 2008.
Y. McLatchie, E. Fong, D. T. Frazier, and J. Knoblauch. Predictive performance of power posteriors. Biometrika, page asaf034, 2025.
F. Wenzel, K. Roth, B. Veeling, J. Swiatkowski, L. Tran, S. Mandt, J. Snoek, T. Salimans, R. Jenatton, and S. Nowozin. How good is the Bayes posterior in deep neural networks really? In International Conference on Machine Learning, pages 10248–10259. PMLR, 2020.
T. Minka. Divergence measures and message passing. https://tminka.github.io/papers/message-passing/, Dec. 2005.
L. Zellinger and A. Vergari. Even more guarantees for variational inference in the presence of symmetries. arXiv preprint arXiv:2604.21407, 2026.
S. Nabarro, S. Ganev, A. Garriga-Alonso, V. Fortuin, M. van der Wilk, and L. Aitchison. Data augmentation in Bayesian neural networks and the cold posterior effect. In Uncertainty in Artificial Intelligence, pages 1434–1444. PMLR, 2022.
C. Zeno, I. Golan, A. Pakman, and D. Soudry. Why cold posteriors? on the suboptimal generalization of optimal Bayes estimates. In Third Symposium on Advances in Approximate Bayesian Inference, 2020.
L. Noci, K. Roth, G. Bachmann, S. Nowozin, and T. Hofmann. Disentangling the roles of curation, dataaugmentation and the prior in the cold posterior effect. Advances in neural information processing systems, 34: 12738–12748, 2021.
Y. Zhang, Y.-S. Wu, L. A. Ortega, and A. R. Masegosa. The cold posterior effect indicates underfitting, and cold posteriors represent a fully Bayesian method to mitigate it. Transactions on Machine Learning Research, 2024.
K. Osawa, S. Swaroop, M. E. Khan, A. Jain, R. Eschenhagen, R. E. Turner, and R. Yokota. Practical deep learning with Bayesian principles. Advances in neural information processing systems, 2019. K. Pitas and J. Arbel. Cold posteriors through PAC-Bayes. In NeurIPS 2022 Workshop on ML Safety, 2022. C. E. Rasmussen and C. K. I. Williams. Gaussian Processes for Machine Learning. The MIT Press, Nov. 2005. ISBN 978-0-262-25683-4. doi: 10.7551/mitpress/3206.001. 0001. L. Sagun, L. Bottou, and Y. LeCun. Eigenvalues of the Hessian in deep learning: Singularity and beyond. arXiv preprint arXiv:1611.07476, 2016. Y. Shen, N. Daheim, B. Cong, P. Nickl, G. M. Marconi, C. Bazan, R. Yokota, I. Gurevych, D. Cremers, M. E. Khan, and T. Möllenhoff. Variational learning is effective for large deep networks. In International Conference on Machine Learning, pages 44665–44686. PMLR, 2024. S. Sun, G. Zhang, J. Shi, and R. Grosse. Functional variational Bayesian neural networks. In International Conference on Learning Representations, 2019. M. Titsias. Variational learning of inducing variables in sparse Gaussian processes. In International Conference on Artificial Intelligence and Statistics, pages 567–574. PMLR, 2009. R. E. Turner and M. Sahani. Two problems with variational expectation maximisation for time series models, pages 104–124. Cambridge University Press, 2011. 11
Gaussian MFVI can overestimate predictive variance
A. Proofs of results A.1. Optimal Variational Parameters In this section we restate and prove the optimal variational parameters. Lemma (Optimal variational mean). For any positive definite posterior covariance, the optimal mean is the mean of the posterior: m∗ = µ. (21) Proof of Lemma 3.1. If Σ is positive definite, then Σ−1 is also positive definite. By definition z ⊤ Σ−1 z ≥ 0,
(22)
with the minimum being achieved when z = 0. This will be achieved when m = µ. Lemma (Optimal variational posterior eigenvalues are harmonic means). The optimal variational covariance, S ∗ , is given by the condition P X wpq 1 ⊤ −1 Σ e = = e ∀ p ∈ {1, . . . P }, (23) p p ∗ dp δq q=1 where
PP
q=1 wpq = 1 ∀ p ∈ {1, . . . , P }.
Proof of Lemma 3.2. Extracting the terms which depend on S from the objective, we are trying to minimise F = Tr Σ−1 S − log det S.
(24)
These terms can be rewritten in terms of the eigen basis and the diagonal covariances of S as P X −1 Tr Σ−1 S = dp e⊤ ep . pΣ
(25)
p=1
Assuming that we restrict dp > 0 log det S =
P X
(26)
log dp .
p=1
Substituting these into F gives F =
P X p=1
−1 dp e⊤ ep − log dp . pΣ
(27)
Taking the derivative wrt dp and setting this equal to zero gives −1 e⊤ ep = pΣ
1 . d∗p
(28)
For the second equality we need to substitute the diagonalised form of the true posterior, so −1 e⊤ ep = pΣ
P X
⊤ −1 e⊤ p vq vq ep δq =
q=1
P X
2 −1 (e⊤ p vq ) δ q . | {z } q=1
(29)
wpq
2 We define wpq = (e⊤ p vq ) , which must sum to one, as it is the definition of the L2 norm squared of the vector ep which is defined to be equal to one. We provide a formal proof of this sum to one criterion Lemma A.1.
12
Gaussian MFVI can overestimate predictive variance P Lemma A.1. For any two orthonormal bases, {ep }P p=1 and {vq }q=1 , the weights defined by the squared inner product ⊤ 2 between the vectors, wpq = (ep vq ) , obey the equality P X
wpq =
P X
p=1
(30)
wpq = 1.
q=1
PP ⊤ Proof. Consider a vector a and any orthonormal basis {zk }P k=1 . The vector a can be written as a = k=1 (a zk )zk , where ⊤ the inner product (a zk ) defines the magnitude of the vector along each of the orthogonal directions of the basis. From this it is clear that the squared L2 distance is given, by definition, as ||a||22 =
P X
(a⊤ zk )2 .
(31)
k=1
P P P Substituting a = ep and {vq }P q=1 = {zk }k=1 , or a = vq and {ep }p=1 = {zk }k=1 give the two desired equalities.
A.2. MFVI predictions underestimate uncertainty for isotropic test points This section gives a detailed proof that the expected predicted variance for the MFVI posterior is less than that of the exact posterior for a test point distributed as x ∼ N (0, IP ).
One way to prove this is to first consider the predictive distribution when the test point is equal to an eigenvector of the MFVI covariance, i.e. at x = ep for any p = 1, . . . , P . At any one of these points, the variational posterior will underestimate the epistemic uncertainty of the exact posterior. Formally, we can state the following Lemma. Lemma A.2 (Variance Underestimation at MFVI basis vectors). For any canonical basis vector ep , p ∈ {1, . . . , P } the predictive variances are governed by ⊤ ∗ e⊤ (32) p Σep ≥ ep S ep . Proof of Lemma A.2. For the true posterior, the predictive variance is given by e⊤ p Σep =
P X
(33)
wpq δq ,
q=1
which is the Arithmetic Mean (AM) of the eigenvalues of the true posterior with weights {wpq }P q=1 . For the MFVI posterior, the predictive variance is given by
∗ e⊤ p S ep = dp = PP
1
−1 q=1 wpq δq
,
(34)
which is the Harmonic Mean (HM) of the eigenvalues of the true posterior with the same weights. The well known AM-HM inequality gives the desired result. By noting that the quadratic form e⊤ p Aep = App , with App being the pth diagonal element of the matrix A, the result above also easily extends to the following inequality for the trace. Lemma A.3 (Trace Inequality.). The trace of the MFVI posterior and the true posterior are governed by Tr (Σ) ≥ Tr (S ∗ ) .
(35)
Proof of Lemma A.3. In the basis of the MFVI posterior, trace is given by Tr (Σ) =
P X
e⊤ p Σep ,
Tr (S ∗ ) =
p=1
P X
∗ e⊤ p S ep .
(36)
p=1
Each of these terms p = 1, . . . , P is governed by the inequality in Lemma A.2, meaning each term in this sum can be bound ⊤ ∗ by e⊤ p Σep ≥ ep S ep , hence the sum of all these terms must also be bound. 13
Gaussian MFVI can overestimate predictive variance
From here, we can directly move to proving the isotropic variance overestimation result. Lemma. For a test point x ∼ N (0, IP ), the predictive variances of the MFVI posterior and exact posterior are governed by Ex∼N (0,IP ) x⊤ Σx ≥ Ex∼N (0,IP ) x⊤ S ∗ x (37) Proof. By noting Ex∼N (0,IP ) x⊤ Σx = Tr (Σ) , Ex∼N (0,IP ) x⊤ S ∗ x = Tr (S ∗ ) ,
(38) (39)
the result follows directly from A.3. A.3. MFVI overestimates uncertainty along the first principal component of the training data Here we provide the proof of Lemma 3.4, with the argument relating this result to the distribution of the training inputs being given in the main text. Lemma. (Overestimation of predictive variance in one direction of the input space.) For points in the input space defined by x̃ ∈ span(vq∗ ) for some scalar, the predictive variance for the exact posterior and the MFVI posterior are governed by the inequality x̃⊤ S ∗ x̃ ≥ x̃⊤ Σx̃. (40) Proof of Lemma 3.4. Let x̃ = cvq∗ , for some constant c. For c = 0 the equality holds trivially. For any non-zero constant c, the predictive variance of the two posteriors are given by x̃⊤ S ∗ x̃ = c2 vq⊤∗ Svq∗
and x̃⊤ Σx̃ = c2 vq⊤∗ Σvq∗ ,
(41)
so proving the result for c = 1 gives the result for all other non zero values of c. The predictive variance of the MFVI posterior is given by P X
vq⊤∗ S ∗ vq∗ = vq⊤∗
! ep e⊤ p dp
v
q∗
P X = (vq⊤∗ ep )2 dp .
p=1
(42)
p=1
Note that (vq⊤∗ ep )2 = wpq∗ which is the weighted term relating the MFVI posterior eigenvalues to the exact posterior eigenvalues. Making this substitution, along with the values of the eigenvalues of dp from Lemma 3.2, we can use vq⊤∗ S ∗ vq∗ =
P X p=1
wpq∗ PP
−1 q=1 wpq δq
.
(43)
As we have defined δq∗ to be the minimum eigenvalue, we can use the inequality vq⊤∗ S ∗ vq∗ ≥
P X p=1
wpq∗ PP
= −1
q=1 wpq δq ∗
P X wpq∗
P X
δq∗
p=1
p=1
−1 =
wpq∗ δq∗ = δq∗ = vq∗ Σvq∗ .
(44)
This proves the statement for c = 1, hence giving the result for all non-zero c.
Following this we can simply show the overestimation of predictive variance along the first principal component. Theorem. Consider the conjugate Bayesian linear model of Equation (1) with a spherical prior θ ∼ N (0, α−1 I), and let w denote the first principal component of the training inputs X. Then for any test point x̃ ∈ span(w), the predictive variances of the MFVI and exact posteriors satisfy x̃⊤ S ∗ x̃ ≥ x̃⊤ Σx̃. (45) 14
Gaussian MFVI can overestimate predictive variance
Proof of Theorem 3.5. For a spherical prior θ ∼ N (0, α−1 I), the exact posterior covariance Σ shares its eigenbasis with the Gram matrix X ⊤ X, since αI commutes with every matrix. Writing this shared eigenbasis as {vq }P q=1 with associated Gram matrix eigenvalues {γq }P , the posterior eigenvalues are q=1 γq −1 . δq = α + 2 σ
(46)
This relationship is strictly decreasing in γq , so the eigenvector associated with the largest Gram matrix eigenvalue is the eigenvector associated with the smallest posterior eigenvalue. The former is, by definition, the first principal component of the training data, w, and the latter is vq∗ as defined in Lemma 3.4, so w = vq∗ . Hence span(w) = span(vq∗ ), and any test point x̃ ∈ span(w) satisfies the hypothesis of Lemma 3.4. Applying that lemma directly gives x̃⊤ S ∗ x̃ ≥ x̃⊤ Σx̃, (47) as claimed. A.4. MFVI under- and overestimates uncertainties similarly The easiest way to show Lemma 3.6 is to first show a particular quantity, Tr Σ−1 S ∗ , is conserved for any exact posterior covariance. Lemma A.4 (Conserved Trace Product). For any eigenbasis of the MFVI posterior, the optimal MFVI posterior covariance will obey Tr Σ−1 S ∗ = P. (48) Proof. The trace of any P × P P matrix A can be calculated by the sum of quadratic forms of the orthonormal eigenbasis of P the MFVI posterior: Tr (A) = p=1 e⊤ p Aep . Applying this to the product of the inverse of the true posterior covariance matrix with the MFVI posterior covariance matrix gives P P P X X X −1 ∗ ∗ ⊤ −1 Tr Σ−1 S ∗ = e⊤ Σ S e = d e Σ e = 1 = P. p p p p p p=1
p=1
(49)
p=1
From this it is easy to show Lemma 3.6. Lemma (Calibrated MFVI). Consider the set of eigenvectors of the true posterior {vq }P q=1 . For these points P
1 X R(vq ) = 1. P q=1
(50)
Proof of Lemma 3.6. For any matrix A and orthonormal basis {vq }P q=1 , we have Tr(A) = −1 ∗ A = Σ S gives P X Tr(Σ−1 S ∗ ) = vq⊤ Σ−1 S ∗ vq .
PP
⊤ q=1 vq Avq . Applying this to
(51)
q=1
Since vq is an eigenvector of Σ with eigenvalue δq , we have vq⊤ Σ−1 = δ1q vq⊤ . Thus Tr(Σ−1 S ∗ ) =
P X vq⊤ S ∗ vq q=1
δq
=
P X vq⊤ S ∗ vq q=1
vq⊤ Σvq
=
From Lemma A.4 this equals P , and dividing both sides by P gives the result. 15
P X q=1
R(vq ).
(52)
Gaussian MFVI can overestimate predictive variance
A.5. MFVI overestimates predictive variance for data with the empirical covariance Theorem. For test data points distributed according to x ∼ p̂(x), the difference in the expected predicted variance is given by Ex∼p̂(x) x⊤ Σx − x⊤ S ∗ x ≤ 0. (53) Proof of Theorem 3.7. Due to the linearity of the expectation, we can write Ex∼p̂(x) x⊤ Σx − x⊤ S ∗ x = Ex∼p̂(x) x⊤ Σx − Ex∼p̂(x) x⊤ S ∗ x ,
(54)
which allows us to deal with the two expectations separately. To prove the result we need to make use of the following identity: Γ̂ =
σ2 Σ−1 − αI . N
By making use of the identity in Equation (55), we can write Ex∼p̂(x) x⊤ Σx = Tr Γ̂Σ 2 σ Σ−1 − αI Σ = Tr N 2 σ = Tr Σ−1 Σ − αTr (Σ) N σ2 (Tr (I) − αTr (Σ)) = N σ2 = (P − αTr (Σ)) . N
(55)
(56)
We can make use of Lemma A.4 to see Ex∼p̂(x) x⊤ S ∗ x = Tr Γ̂S ∗ 2 σ = Tr Σ−1 − αI S ∗ N 2 σ = Tr Σ−1 S ∗ − αTr (S ∗ ) N σ2 = (P − αTr (S ∗ )) . N
(57)
Subtracting one of these from the other gives σ2 α Ex∼p̂(x) x⊤ Σx − Ex∼p̂(x) x⊤ S ∗ x = (Tr (S ∗ ) − Tr (Σ)) . N
(58)
2
As σNα > 0, this value will take the same sign as the difference in the traces. From Lemma A.3 we know that Tr (S ∗ ) − Tr (Σ) ≤ 0,
(59)
hence the expected posterior predictive variance of the true posterior is less than the expected posterior predictive variance of the MFVI posterior. A.6. Cold Posteriors MFVI posterior rescaling for cold posterior. Here, we prove that the cold posterior MFVI objective has the form qT (θ) = N (m∗ , T S ∗ ) as stated in Section 4, where m∗ and S ∗ are the solutions at T = 1 given in Equations (6) and (7). 16
Gaussian MFVI can overestimate predictive variance
Proof. In the conjugate case with prior N (0, α−1 IP ) and likelihood N (y|θ⊤ x, σ 2 ), the cold posterior is given by "N # Y 1 1 pT (θ|D) ∝ N (yn |θ⊤ xn , σ 2 ) T N (θ|0, α−1 I) T
(60)
n=1
1# α T1 2 T 1 ⊤ ∝ exp − 2 yn − θ xn exp − θ2 2σ 2 n=1 "N # α Y 2 1 ⊤ = y − θ x exp − exp − θ2 n n 2 2T σ 2T n=1 "N # Y T ∝ N (yn |θ⊤ xn , T σ 2 ) N θ|0, IP . α n=1 "N Y
(61) (62) (63)
In other words, this is the exact posterior for a system with a rescaled likelihood and prior. Applying the known results for the covariance gives −1 α 1 ⊤ IP + X X T T σ2 −1 1 ⊤ = T αIP + 2 X X σ
ΣT =
(64)
= T Σ, so the exact cold-posterior covariance is the exact posterior at T = 1 scaled by a factor T. Applying the known result for the mean gives µT = ΣT
1 X ⊤Y T σ2
=
1 ΣX ⊤ Y = m, σ2
(65)
so applying the exponentiation does not change the mean of the posterior. From here it is straightforward to apply the optimal variational parameter arguments to get mT = m = µ ∀ T,
(66)
so the change in temperature does not effect the mean values. Similarly, dpT = so
1 T = ⊤ −1 = T dp , 1 −1 e Σ ep e⊤ Σ e p p p T ST = T S ∗ .
(67) (68)
A.7. Pathological example: Low Rank Linear Regression Proposition. As P → ∞, for any test input x ∈ RP the posterior predictive variance of the MFVI posterior will be given by x⊤ S ∗ x = α−1 ||x||22 ,
(69)
where α is the prior precision. Proof of Proposition 5.1. We start this proof by considering the symmetries, thereby giving us the weights which relate the eigenvalues of the exact posterior to the MFVI posterior. As the relation between 1P and each canonical basis function 17
Gaussian MFVI can overestimate predictive variance
is identical, so it is clear that all values of wpq are the same. Combining this with the sum-to-one requirement gives wpq = P1 ∀ p, q.
Next we note that in the exact posterior all of the eigenvalues will be α−1 , apart from the one associated with the eigenvector which points towards the span of the data, which we will denote by δmin . x We can then consider the predicted variance which would be given by the normalised test point ||x|| 2 this into the predicted 2 variance gives P x⊤ S ∗ x X wpq = PP wpq ||x||22 p=1 q=1
(70)
δq
=
P X p=1
= =
1 P
(71)
PP
1 q=1 P δq
P X
1 P 1 + (P − 1)α p=1 P δmin
1 (P −1)α 1 P δmin + P
(72) (73)
Inspecting the denominator 1 1 (P − 1)α α = + − + α. P δmin P P δmin P
(74)
As P → ∞ the first two terms go to zero, leaving α. Applying the limit quotient rule we see x⊤ S ∗ x = α−1 ||x||22
(75)
x⊤ S ∗ x = α−1 ||x||22 .
(76)
and hence
B. Experiment Details Pre-processing and Train-Test Split. Given a full data set D = (X, Y ) as defined in Section 3, we make a split into a train, ID test and OOD test set as follows. First, the OOD test set is formed by sorting and then splitting the data on a chosen feature. The remainder is randomly split into a train and an ID test set. All inputs are then standardized using the train set mean and standard deviation, and the outputs are all centred with the train set output mean. The random splits are independent across trials, giving the desired variability in train-test splits, as well as learned hyperparameters (see below). Basis functions.
All experiments use RBF basis functions to construct the feature map, φ : RP 7→ RQ , with ∥x−cq ∥22 Q φ(x) = e− 2l , q=1
(77)
P for basis centroids {cq }Q q=1 where cq ∈ R and a common lengthscale, l. The centroids are sampled independently as
cq ∼ N (0P , M ),
(78)
1 ⊤ with M = ntrain Xtrain Xtrain , the empirical covariance of the standardized training inputs. We denote the inputs, X, mapped to this basis by Φl , where the subscript makes explicit the dependence on the learnable lengthscale.
18
Gaussian MFVI can overestimate predictive variance
Learning hyperparameters. The three learnable hyperparameters in our experiments are the observation noise, σ, the prior precision, α, and the RBF lengthscale, l. They are chosen by gradient-based optimization of the train set marginal likelihood, given by 2 p(Ytrain | Xtrain ) = N (0ntrain , α−1 Φl,train Φ⊤ l,train + σ Intrain ),
(79)
and, for numerical stability, we minimise the negative log likelihood, (80)
L(σ, α, l) = − log p(Ytrain | Xtrain ). To this end, we use the ADAM optimizer (Kingma and Ba, 2015) with default settings for a fixed 2000 gradient steps.
Posterior predictive. Both the true and T -MFVI posterior predictive considered in our experiments are analytically tractable in the conjugate BLR setting. We distinguish the joint posterior predictive for a test set, (X, Y ) and the marginal one, for a test point, (x, y). The marginal true posterior predictive at an input, x, is given by Z p(y | x, D) = p(y | θ, x) p(θ | D) dθ (81) = N (y; µ⊤ x, x⊤ Σx + σ 2 ),
(82)
with µ and Σ from Equation (2). The marginal T -MFVI posterior predictive is Z qT (y | x, D) = p(y | θ, x) qT (θ | D) dθ
(83)
= N (y; m∗ ⊤ x, T x⊤ S ∗ x + σ 2 ),
(84)
with m∗ and S ∗ from Equations (6) and (7). For the joint posterior predictives, we have Z p(Y | X, D) = p(Y | θ, X) p(θ | D) dθ
(85)
= N (Y ; Xµ, XΣX ⊤ + σ 2 I),
(86)
Z
p(Y | θ, X) qT (θ | D) dθ
(87)
= N (y; Xm∗ , T XS ∗ X ⊤ + σ 2 I).
(88)
and qT (Y | X, D) =
Divergences. A number of divergences are considered in Figure 6, with the aim of quantifying how close the T -MFVI posterior predictive is to the true posterior predictive. While we write the divergences for the marginal posterior predictives here, they straightforwardly extend to the joint predictive setting. We use both the forward and reverse Kullback-Leibler (KL) divergence, Z p(y, | x, D) DKL (p∥qT ) = p(y, | x, D) log dy, (89) qT (y, | x, D) Z qT (y, | x, D) DKL (qT ∥p) = qT (y, | x, D) log dy. (90) p(y, | x, D) (91) We also consider the standard α-divergence at α = 0.5, Dα (p, qT ) = 4 1 −
Z p
! p(y, | x, D)qT (y, | x, D) dy , 19
(92)
Gaussian MFVI can overestimate predictive variance
making it proportional to Hellingers distance, and thus symmetric. Further, we make use of the 2-Wasserstein distance, which is analytic for two Gaussians and given by W2 (p, qT ) =
1
inf
γ∈Γ(p,qT )
E(y,y′ )∼γ [∥y − y ′ ∥2 ] 2 ,
(93)
with Γ(p, qT ) the set of all couplings of the two posterior predictive distributions. Lastly, for comparing marginal predictives, we consider the simple squared difference between predictive variances (σp2 − σq2T )2 ≡ (x⊤ S ∗ x + σ 2 − x⊤ Σx − σ 2 )2 ⊤
∗
⊤
2
= (x S x − x Σx) ,
(94) (95)
which we generalize to the squared Frobenius norm ∥Σp − ΣqT ∥2F ≡ ∥XΣX ⊤ + σ 2 I − T XS ∗ X ⊤ − σ 2 I∥2F ⊤
= ∥XΣX − T XS
∗
X ⊤ ∥2F ,
(96) (97)
for comparing joint predictives. When considering the marginal posterior predictives for a given divergence, the marginal divergences are simply averaged across test points, reflecting a marginal per-data-point predictive divergence. For the joint posterior predictves, each test set simply gives one divergence between the two multivariate Gaussians. This is done for a grid of 100 values T (equally spaced on the log scale), and T ∗ in Figure 6 is then found as the divergence minimiser on this grid.
C. Additional Experiment Results Here we present experimental results following the set up of Appendix B and Section 5.2, for additional UCI data sets. Analogous to Figure 6, Figures 8 and 9 show the temperature minimising various measures of divergence between the true posterior predictive and the T −MFVI posterior predictive.
20
Gaussian MFVI can overestimate predictive variance
boston
concrete
Marginal divergences
Marginal divergences −0.2
−0.8
log10 (T ∗ )
log10 (T ∗ )
−0.7
−0.9 −1.0
−0.4 −0.6
−1.1
−0.8
−1.2
−1.0
Joint divergences
Joint divergences 0.5
0.0
−0.5
log10 (T ∗ )
log10 (T ∗ )
0.0
−1.0
−1.0
−1.5
−1.5
−2.0
−2.0
DKL
(p |
2
) ) | p) qT ) kF p, q T p | qT (q T W 2( − Σq T Dα( DKL kΣ p Train
ID Test
DKL
(p |
OOD Test
2
) ) | p) qT ) kF p, q T p | qT (q T W 2( − Σq T Dα( DKL kΣ p Train
ID Test
energy
power
Marginal divergences
Marginal divergences
OOD Test
0.0
−0.2
−0.2
−0.4
log10 (T ∗ )
log10 (T ∗ )
−0.5
−0.6 −0.8
−0.4 −0.6 −0.8 −1.0
−1.0
−1.2
Joint divergences 0.5 0.0
0.0
log10 (T ∗ )
log10 (T ∗ )
Joint divergences
0.5
−0.5 −1.0 −1.5
−0.5 −1.0 −1.5 −2.0
−2.0
DKL
(p |
qT ) DKL
(q T
Train
| p)
Dα(
p|
qT )
ID Test
W 2(
p,
qT ) kΣ p
− Σq T
2 kF
DKL
(p |
2 ) ) | p) qT ) kF p, q T p | qT (q T W 2( − Σq T Dα( DKL kΣ p
Train
OOD Test
ID Test
OOD Test
Figure 8. Optimal temperatures for various measures of divergence are shown across independent train-test splits.
21
Gaussian MFVI can overestimate predictive variance
protein
wine
Marginal divergences
Marginal divergences −0.4
log10 (T ∗ )
log10 (T ∗ )
−0.2 −0.4 −0.6 −0.8
−0.6 −0.8 −1.0 −1.2
Joint divergences
Joint divergences 0.5
0.0
log10 (T ∗ )
log10 (T ∗ )
0.0 −0.5 −1.0
−0.5 −1.0 −1.5
−1.5
−2.0
DKL
(p |
2
) ) | p) qT ) kF p, q T p | qT (q T W 2( − Σq T Dα( DKL kΣ p Train
ID Test
DKL
OOD Test
2
) ) | p) qT ) kF p, q T p | qT (q T W 2( − Σq T Dα( DKL kΣ p Train
ID Test
yacht
naval
Marginal divergences
Marginal divergences
0.25
OOD Test
1.00
0.00
0.75
−0.25
0.50
log10 (T ∗ )
log10 (T ∗ )
(p |
−0.50 −0.75 −1.00
0.25 0.00 −0.25 −0.50
−1.25
−0.75
Joint divergences
Joint divergences
0.5 0.0
log10 (T ∗ )
log10 (T ∗ )
0.0 −0.5 −1.0 −1.5
−0.5 −1.0 −1.5 −2.0
−2.0
DKL
(p |
qT ) DKL
(q T
Train
| p)
Dα(
p|
qT )
ID Test
W 2(
p,
qT ) kΣ p
− Σq T
2 kF
DKL
OOD Test
(p |
2 ) ) | p) qT ) kF p | q T W 2 (p, q T (q T − Σq T Dα( DKL kΣ p
Train
ID Test
OOD Test
Figure 9. Optimal temperatures for various measures of divergence are shown across independent train-test splits.
22