A Unified Risk View of Uncertainty: Posterior Risk for Disentanglement and Evaluation Beyond Proxies

A Unified Risk View of Uncertainty: Posterior Risk for Disentanglement and Evaluation Beyond Proxies

arXiv:2608.05995v1 [cs.LG] 6 Aug 2026

Frieder Wizgall∗, 1 Georg Tirpitz∗, 1 Moritz Seiler2, 5 Kerstin Ritter2, 3, 4, 5 Bálint Mucsányi1 1 University of Tübingen, Tübingen, Germany, 2 Hertie Institute for AI in Brain Health, University of Tübingen, Tübingen, Germany, 3 Tübingen AI Center,University of Tübingen, Tübingen, Germany, 4 Charité–Universitätsmedizin Berlin, Department of Psychiatry and Psychotherapy, Berlin, Germany, 5 Bernstein Center for Computational Neuroscience Berlin, Berlin, Germany

Abstract Reliable uncertainty estimates are critical in safety-sensitive applications, where understanding the sources of predictive uncertainty is essential. This often requires disentangling epistemic uncertainty from aleatoric uncertainty, yet these uncertainty types are not defined consistently across the literature, making it difficult to assess whether a method produces accurate uncertainty estimates. Evaluation is further complicated by the fact that ground-truth epistemic uncertainty is typically unavailable. Existing benchmarks therefore mostly rely on proxy tasks such as out-of-distribution detection, which do not provide complete ground-truth uncertainty targets and offer limited insight into the structure and quality of uncertainty estimates. We propose a unified definition of uncertainty as pointwise posterior risk, the expected loss of a predictor under the distribution of plausible ground-truth functions given the data. This view combines Bayesian uncertainty over functions with estimator-dependent deviations from the posterior mean, capturing effects such as misspecification and optimization error. This formulation constitutes the foundation of a theory-backed benchmark that enables direct computation of oracle epistemic and aleatoric uncertainty using semi-synthetic datasets with real covariates and known generative processes. By avoiding proxy evaluations, the benchmark enables fine-grained analysis of uncertainty estimates. Empirically, we find that accurate prediction does not guarantee reliable uncertainty disentanglement. The benchmark reveals practically useful differences between methods, identifying approaches with meaningful alignment to oracle uncertainty targets while exposing sensitivity to datasets and modeling choices.

1

Introduction

AI systems are increasingly deployed in safety-critical domains where uncertainty is essential for safety and interpretability rather than optional. In such settings, uncertainty estimates can guide referral to human experts, trigger additional tests, inform additional data acquisition, and generally help decide when a model’s output can be trusted [Snoek et al., 2019, Kompa et al., 2021, Kendall and Gal, 2017]. Crucially, why a model is uncertain matters as much as how much: uncertainty can be irreducible due to intrinsic randomness (aleatoric uncertainty/AU), or reducible because the model is poorly optimized, lacks coverage, or is poorly specified (epistemic uncertainty/EU) [Hüllermeier * Equal contribution (shared first authorship). Preprint.

and Waegeman, 2021]. This distinction is widely advocated, for example, in applied clinical ML, where uncertainty is used to flag unreliable cases for review [Faghani et al., 2023, Huang et al., 2024]. For these flags to be useful, uncertainty estimates must be reliable, which remains insufficiently established. Bayesian and frequentist concepts of uncertainty. Many approaches to perform uncertainty disentanglement (UD) have been proposed, but no unified view has emerged. Bayesian approaches model uncertainty via a distribution over possible ground-truth functions given the observed data, whereas frequentist notions quantify uncertainty as the expected risk of an estimator with respect to a fixed ground-truth function. While both perspectives capture important aspects of uncertainty, they rely on strong and often implicit assumptions. Risk-based Frequentist formulations assume uncertainty to be the error of a single fixed data-generating function, which can lead to unrealistically low uncertainty in regions with limited data coverage. In contrast, standard Bayesian uncertainty measures typically quantify uncertainty within the assumed model or prior, and therefore do not directly capture estimator-dependent effects such as misspecification or suboptimal optimization. We show that both views arise as special cases of a more general notion of uncertainty. Specifically, we define uncertainty as sample-conditional posterior risk: the expected loss of a predictor under the distribution of plausible ground-truth functions given the observed data. This perspective naturally combines uncertainty over functions with estimator-dependent effects, and recovers standard Bayesian and frequentist formulations under additional assumptions, which we make explicit in the following. Methods and benchmarking of UD. The divergent definitions of uncertainty, together with the intrinsic difficulty of quantifying it in the absence of ground truth, have led to a wide range of proposed methods for UD Kotelevskii et al. [2024], Lahlou et al. [2021]. This raises the question of which methods are most reliable and accurate in different machine learning scenarios. While some efforts have been made to determine this, we see two main issues in the UD benchmarking community: First, there is no unified benchmark for UD. Existing benchmarks use different datasets, tasks, and definitions, making comparison difficult Kotelevskii et al. [2024], Fishkov et al. [2025], Lahlou et al. [2021], Jiménez et al. [2025], Mucsányi et al. [2024]. Unified benchmarks can serve as major drivers of progress in a field, as demonstrated in other disciplines of machine learning Russakovsky et al. [2015], by enabling direct comparison of methods on the same task and data. Second, defining an oracle EU is difficult Jürgens et al. [2024], Jiménez et al. [2025], which led most benchmarking efforts to rely on proxy tasks like out-of-distribution (OOD) detection or correlations between disentangled uncertainty components Mucsányi et al. [2024]. While these can give insight into the quality of UD, OOD detection is generally not a reliable ground truth for epistemic uncertainty: the objectives are fundamentally different, and even perfect epistemic uncertainty would not suffice to solve OOD detection in general [Li et al., 2025]. Empirically, strong OOD performance can coexist with poorly calibrated epistemic uncertainty under distributional shift [Postels et al., 2022]. Other analyses further highlight the conceptual mismatch (e.g., predictive entropy conflating ambiguous indistribution inputs with OOD) [Kirsch et al., 2021]. We see the need to define a notion of uncertainty disentanglement that can be compared directly with model outputs and that allows for detailed testing and deeper analysis. It should also satisfy the intuitions of what EU should represent to enable more precise and faithful benchmarking. This is especially important in safety-critical settings, where practitioners need uncertainty estimates they can trust, as well as tools to diagnose when and why they fail. Semi-synthetic evaluation under a tractable posterior. To make the proposed sample-conditional posterior risk notion operational, we need a setting in which the sample-conditional posterior over ground-truth functions is explicit and samplable. We therefore instantiate the theory in a semisynthetic tabular regression setting with real covariates and synthetic targets drawn from a Gaussianprocess prior. We do not claim that this GP-based construction provides a notion of ground truth that generalizes to all other modalities; rather, it gives a controlled reference setting in which estimatorspecific oracle targets for aleatoric and epistemic uncertainty can be computed exactly under the assumed generative process. This enables a much more fine-grained analysis than proxy evaluations, and allows us to study how UD is affected not only by the assumed posterior semantics, but also by distributional assumptions and design choices throughout the modeling pipeline. This GP restriction is specific to the evaluation protocol and does not limit the generality of the sample-conditional posterior-risk definition itself. To summarize, we make the following contributions: 2

Figure 1: Different definitions of epistemic uncertainty. Exc. is the abbreviation for Excess. A: Epistemic uncertainty defined as the Bayesian posterior variance of the true posterior. B: EU defined as pointwise excess risk. C: Oracle EU according to the resampling frequentist view. D: Our pointwise posterior excess risk oracle EU. Substantial differences between the definitions are visible, especially in how they behave away from the observations and under estimator misspecification. • A unified definition of uncertainty as sample-conditional posterior risk. We define sample-conditional pointwise posterior risk as a rigorous notion of uncertainty, capturing the expected loss of a predictor under the distribution of plausible ground-truth functions and making explicit how Bayesian and frequentist views arise as special cases. • A fine-grained empirical study of uncertainty disentanglement under a tractable posterior. We conduct an extensive evaluation on GP-generated semi-synthetic regression tasks with real covariates, enabling direct comparison of epistemic and aleatoric uncertainty estimates against estimator-specific posterior-risk targets and allowing us to analyze the effects of model class, architecture, activation, optimization, and validation choices beyond what proxy tasks can reveal.

2

Uncertainty and pointwise risk

2.1

Variables and Notation

We consider supervised learning with an unknown joint distribution P over (X, Y ) ∈ X × Y, where X ⊆ Rd and Y ⊆ R. Given x ∈ X , let Px := P ( · | X=x) denote the conditional distribution of Y and write p(y | x) for its density. Let S := {(Xi , Yi )}N i=1 denote the (random) training dataset i.i.d.

with (Xi , Yi ) ∼ P . We write S := {(xi , yi )}N i=1 for its realized value (the observed sample). A learned predictor fˆ maps inputs to a target space Y, i.e. fˆ : X → Y. Depending on the setting, fˆ(x) is either a point prediction in Y or a predictive distribution in P(Y). We use a single loss symbol ℓ : A × Y → R, covering both pointwise losses and proper scoring rules. We write ℓ(Q, y) for a (strictly) proper loss. 2.2

Uncertainty via risk

Following the pointwise risk view of uncertainty Lahlou et al. [2021], Kotelevskii et al. [2022, 2024], Fishkov et al. [2025]: Definition 2.1 (The pointwise risk view). Let A denote the action space of the predictor. The pointwise risk of fˆ at x is Z  
R(fˆ; x) := EY ∼Px ℓ(fˆ(x), Y ) = ℓ fˆ(x), y p(y | x) dy. (1)   For each x, the optimal (Bayes) pointwise risk is R⋆ (x) := inf a∈A EY ∼Px ℓ(a, Y ) . A minimizer a⋆ (x) is called a Bayes act. When ℓ is a strictly proper scoring rule over predictive distributions, this Bayes act is uniquely Px ; for ordinary point losses it is the corresponding optimal point prediction, such as the conditional mean under squared loss.

3

Implication 1 The definition supports covariate shift (“Soft-OOD”), but it is not invariant to distributional changes in Px and does not address strict distribution-membership detection (OOD). “Soft-OOD” is the regime in which risk-based epistemic uncertainty is comparable [Kotelevskii et al., 2024]. 2.3

Epistemic uncertainty is an estimator’s risk over the distribution of ground-truth functions.

We propose adopting the learner’s perspective for a more intuitive UD: We are given a set of samples S, and aim to accurately determine the ground-truth function f that generated S. We now argue from a Bayesian perspective that the uncertainty represents the risk of our predictor fˆ(x) under the distribution of possible ground-truth functions. Even if one believes there exists a single “true” function f ⋆ , the information available to the learner is the finite sample S which could have been generated by a whole distribution of plausible ground-truth functions. A convenient way to formalize this is to introduce a latent random function f and reason about its conditional distribution given S. Definition 2.2 (Sample-conditional pointwise posterior risk). For a fixed input x and observed sample S, the pointwise posterior risk of a deployed predictor fˆS is Z  
RS (fˆS ; x) := EY ∼P (·|x,S) ℓ(fˆS (x), Y ) = ℓ fˆS (x), y p(y | x, S) dy ZZ (2) = ℓ(fˆS (x), y) p(y | x, f ) Π(df | S) dy   = Ef ∼Π(·|S) EY ∼P (·|x,f ) ℓ(fˆS (x), Y ) Here, Π(· | S) is the posterior distribution over latent functions f given the observed sample S (when a density exists, Π(df | S) = p(f | S) df ). Conditioned on (x, f ), the outcome follows the noise model R Y ∼ P (· | x, f ) with density p(y | x, f R). The posterior predictive is the mixture P (· | x, S) = P (· | x, f ) Π(df | S), and p(y | x, S) = p(y | x, f ) Π(df | S). For notational convenience, when x and S are fixed we will write Ef [·] := Ef ∼Π(·|S) [·], EY |f [·] := EY ∼P (·|x,f ) [·], so that RS (fˆS ; x) = Ef EY |f ℓ(fˆS (x), Y ).

Similar to [Adlam et al., 2022], we use the Bregman divergence framework of [Pfau, 2025] to decompose this into AU and different sources of EU to get the most general notion of disentanglement. Following our definition in Equation (2), we derive the decomposition by taking the expectation over the latent functions via p(f | S) (instead of data samples with p(S)). We take the expectation pointwise in x and keep S fixed. In this notation, UY denotes the representation of the observation Y in the action space on which the Bregman divergence is defined. For squared loss, UY = Y ; for scores such as CRPS, UY can be viewed as the indicator function t 7→ 1{Y ≤ t}. Proposition 2.3 (Bregman decomposition of posterior risk). Let D be a convex action space and let ϕ : D → R be strictly convex and continuously differentiable. Let Dϕ denote the Bregman divergence induced by ϕ. Assume all expectations below are finite and that the minimizers are well-defined. Applying the Bregman bias–variance identity [Pfau, 2025] pointwise in x, the pointwise posterior risk decomposes as h h  Ef EY |x,f Dϕ UY

h  i   = Ef EY |x,f [Dϕ (UY ∥ af (x))] + Ef Dϕ af (x) fˆS (x) | {z } | {z } AU EU     = Ef EY |x,f [Dϕ (UY ∥ af (x))] + Ef [Dϕ (af (x) ∥ ā(x))] + Dϕ ā(x) fˆS (x) {z } | {z } | | {z } fˆS (x)

Bayes error

ii

generalized variance

generalized bias

with af (x) ∈ arg min EY |x,f [Dϕ (UY ∥ z)] ,

ā(x) ∈ arg min Ef |S [Dϕ (af (x) ∥ z)] .

z

z

4

(3)

Here af (x) is the conditional Bayes act and ā(x) the corresponding Bregman centroid, so epistemic uncertainty decomposes into generalized variance and generalized bias. We observe that in Equation (3) we can plug in the respective strictly proper losses/scores to yield specific variants of UD. For example, under the MSE loss with additive Gaussian noise, the conditional Bayes act reduces to the latent function value, af (x) = f (x), and the centroid becomes ā(x) = mS (x), yielding (more detailed version of Equation (3) and examples of other losses/scores in Appendix A):
2 RS (fˆ; x) = σϵ2 (x) + σf2 (x) + fˆ(x) − mS (x) | {z } | {z } | {z } AU Bias Post. variance | {z }

(4)

EU

Implication 2 The semantics of AU and EU are determined by the loss or proper scoring rule used in the oracle risk evaluation. This should not be confused with the learner’s training objective, which may be different. The benchmark evaluates uncertainty through the posterior-risk target, not through the loss used to train the estimator. 2.4

Sample-Conditional Posterior Risk as a Compromise Between Existing Views

The posterior-spread Bayesian view. A common Bayesian view identifies epistemic uncertainty with posterior spread, such as posterior variance under squared loss. In our framework, this view is recovered by assuming that the deployed predictor is the posterior Bayes act under the same posterior used to quantify uncertainty. In the MSE specialization of Equation (4), this means fˆ(x) = mS (x) = Ef |S [f (x)]. Then the estimator-dependent bias term vanishes and RS (mS ; x) = σϵ2 (x) + σf2 (x),

and hence

EUS (mS ; x) = σf2 (x),

recovering the familiar law-of-total-variance structure. This makes explicit the assumption behind posterior-variance-based EU: the deployed predictor is already aligned with the posterior uncertainty model. This assumption is often unrealistic in practice. If the predictor is misspecified, capacitylimited, regularized, or suboptimally optimized, it can be confidently biased even in regions where the model class cannot represent the data-generating function. In such cases, posterior spread alone can underestimate epistemic uncertainty, and model bias may instead be absorbed into the aleatoric component. See Figure 1 (A). The pointwise excess-risk frequentist view. A common frequentist view identifies epistemic uncertainty with pointwise excess risk relative to a fixed ground-truth function. In our framework, this view is recovered by assuming that the learner’s posterior over latent functions has collapsed to a Dirac measure Π(· | S) = δf ⋆ . Then the posterior-spread term in Equation (3) vanishes. In the MSE specialization of Equation (4), this gives
2 mS (x) = f ⋆ (x), σf2 (x) = 0, and hence EUS (fˆ; x) = fˆ(x) − f ⋆ (x) , which is exactly the pointwise excess risk under squared loss. This makes explicit the assumption behind excess-risk-based EU: the learner no longer has posterior uncertainty over which ground-truth functions remain plausible. From the learner’s perspective, this is also restrictive. In regions with little or no nearby training data, many functions may remain compatible with the observations, so uncertainty should remain high even if the deployed predictor happens to be locally close to the realized ground-truth function. Pointwise excess risk can therefore assign low epistemic uncertainty precisely where a finite-sample learner should be cautious. See Figure 1 (B). The resampling frequentist view. The resampling frequentist view takes an outer expectation over repeated dataset draws S ∼ P n under a fixed ground-truth f ⋆ , yielding a reference distribution of first-order predictors under repeated training [Jürgens et al., 2024, Jiménez et al., 2025]. This also supports a (generalized) bias-variance analysis of the learning procedure, but it addresses a different inferential question than in Equation (3): it quantifies how the trained predictor would vary if we 5

could re-collect data and retrain. Additionally, it entails the same notion of local divergence w.r.t. the ground truth as the pointwise frequentist view. This may lead to similar behavior as can be seen in Figure 1 (C). Sample-conditional pointwise posterior excess risk. We condition on the observed dataset S and evaluate the deployed predictor under the posterior over plausible ground-truth functions f ∼ p(f | S). The resulting pointwise posterior risk quantifies total uncertainty, and its excess over the function-conditional Bayes risk defines epistemic uncertainty. We view this as a compromise between two incomplete endpoints. If one measures only posterior spread, epistemic uncertainty ignores estimator-dependent effects such as misspecification and optimization error. If one instead relies only on pointwise excess risk relative to a single realized ground-truth function, epistemic uncertainty is driven entirely by local prediction error and can become too tightly coupled to the ability of the deployed model to fit that particular realization. Our definition retains both ingredients: posterior spread over plausible functions and estimator-dependent bias of the deployed predictor. From the learner’s finite-sample perspective, both are necessary. Limited coverage should leave uncertainty over which functions remain plausible, while misspecification or optimization error should contribute because the deployed predictor may still be systematically wrong even with many observations. In this sense, the resulting epistemic uncertainty is intentionally model-specific, but not reducible to the bias term alone. Rather, it measures the reducible risk faced by the predictor actually deployed after observing S, while avoiding the over-reliance on either posterior spread alone or local error to a fixed realized function. See Figure 1 (D).

3

Related Work

Uncertainty disentanglement and formalization. Predictive uncertainty is commonly separated into aleatoric and epistemic sources [Hüllermeier and Waegeman, 2021, Kendall and Gal, 2017]. In Bayesian and approximate-Bayesian deep learning, this split is often instantiated through informationtheoretic quantities and entropy- or variance-based decompositions [Houlsby et al., 2011, Kendall and Gal, 2017, Smith and Gal, 2018, Charpentier et al., 2022]. Recent axiomatic and empirical work, however, suggests that widely used estimators can violate basic desiderata or fail to disentangle reliably in practice [Wimmer et al., 2022, Bülte et al., 2025, Mucsányi et al., 2024]. Benchmarking, proxy tasks, and misspecification. Because epistemic ground truth is typically unavailable, evaluation often relies on proxy tasks such as OOD detection, or calibration-style stress tests. Yet these objectives can be misaligned with epistemic uncertainty [Li et al., 2025], and entropy-based scores can conflate ambiguity with shift [Kirsch et al., 2021]. Empirically, strong OOD performance can coexist with poorly calibrated epistemic uncertainty under distributional shift [Postels et al., 2022, Gustafsson et al., 2023]. Regression-oriented evaluations therefore often assess uncertainty only indirectly, for example, through controlled simulator studies [Caldeira and Nord, 2021], disentanglement coupling criteria such as low correlation between estimated aleatoric and epistemic components [Valdenegro-Toro and Saromo, 2022], or synthetic references such as QUAM, which compares common approximations against HMC-based posterior predictive uncertainty and reports systematic underestimation by popular methods [Schweighofer et al., 2023]. A further confounder is misspecification, which can suppress epistemic uncertainty or leak model error into aleatoric estimates [Kato et al., 2022, Jiménez et al., 2025], and is also well studied in Gaussianprocess uncertainty disentanglement [Karvonen et al., 2020, Karvonen and Bachoc, 2025].

4

Evaluation protocol

4.1

Semi-synthetic datasets

Oracle aleatoric uncertainty requires the data-generating noise model, while oracle epistemic uncertainty under the posterior-risk view requires the sample-conditional posterior over latent functions p(f | S). Since neither is available for ordinary real datasets, this evaluation benchmark uses a controlled semi-synthetic construction that preserves realistic covariate structure while making oracle uncertainty targets computable. We keep covariates from real UCI tabular regression datasets and generate targets from a known Gaussian-process prior with controlled heteroscedastic noise. The GP kernel hyperparameters are fitted by marginal likelihood optimization on the original regression task 6

before sampling synthetic functions, so the sampled targets inherit realistic smoothness and scale while the posterior and noise model remain known. Given the full covariate pool X = {xi }N i=1 , we sample a latent function f , generate observations yi = f (xi ) + ϵi with ϵi ∼ N (0, σ 2 (xi )), and define σ(xi ) as a linear function of the average feature rank of xi to induce structured heteroscedasticity. A training sample S with n ≪ N is then selected according to a prespecified sampling profile, allowing us to control the amount and type of covariate shift between the training sample and evaluation pool. Conditioning the same GP prior on S yields an analytical, correctly specified posterior p(f | S). Consequently, the posterior-risk oracle targets for aleatoric and epistemic uncertainty are computable at every evaluation point. We provide 14 kernel presets to generate semi-synthetic regression datasets covering different input dimensions, kernel families, noise levels, and covariate-shift profiles with training sample sizes of 750 or 1500 samples. We provide code and scripts to reproduce the experimental pipeline and main benchmark results; full specifications are given in Appendix C. 4.2

Evaluation metrics

We evaluate each reported quantity against its corresponding oracle target. For each method, we compare the predictive mean, aleatoric uncertainty, and epistemic uncertainty using mean squared error (MSE), Pearson correlation r, and Spearman rank correlation ρS over evaluation points. Our primary metric is Spearman rank correlation. We use rank correlation because it provides a meaningful oracle ordering of pointwise uncertainty, while the absolute scale may vary across losses, parameterizations, and model families. Thus, ρS measures whether a method assigns higher uncertainty to truly more uncertain inputs, independent of exact calibration. We also provide the risk coverage and uncertainty calibration curves for all methods.

5

Evaluation

We evaluate uncertainty disentanglement methods in two stages. First, we use a development suite of seven semi-synthetic datasets to select experimental design choices that can strongly affect uncertainty quality, such as neural network architecture, validation objective, and optimizer. Second, after these choices are fixed, we evaluate the resulting methods on a held-out benchmark suite of seven datasets. The held-out suite is used only for the final comparison of methods. This separation is important because, as we will see, the uncertainty estimates are highly sensitive not only to the method family, but also to implementation and training choices. We therefore evaluate these choices in focused ablations to derive practical implementation guidance, and then perform a final benchmark to obtain faithful model rankings on held-out datasets. We evaluate four broad method families: 1. Linear Bayesian baseline: Bayesian linear regression (BLR). 2. Deep kernel learning: deep kernel learning (DKL). 3. Probabilistic tree methods: NGBoost and CatBoost variants. 4. Neural network methods: evidential models, bagging, deep ensembles, dropout, Laplace, FSP-Laplace, SWAG, and DEUP. The final evaluation compares the optimized methods across seven held-out datasets and reports how well each method ranks predictive accuracy, aleatoric uncertainty, and epistemic uncertainty against the corresponding oracle targets, giving practical guidance on which methods perform most robustly across datasets. 5.1

Architecture and Training Configuration Ablations

We conducted five ablation studies on the development dataset suite: width, depth, activation function, tuning objective and optimizer choice, to fix the configurations used throughout the main benchmark. Full results and their more detailed discussions are in Appendix E. 7

Architecture. Prediction quality is largely unaffected by network width and depth, but uncertainty ranking is more sensitive. For width, the clearest pattern appears in aleatoric uncertainty, where 91% of model–dataset pairs show a negative OLS slope of Spearman ρ with oracle uncertainty as width increases. At the same time, the magnitudes of these changes are generally small, and the estimates are noisy across datasets, so we do not interpret this as a strong monotonic law. Rather, the ablation suggests that wider networks should be treated with some caution when uncertainty quality is the primary objective. We therefore restrict width tuning to {16, 32, 64}. Depth shows an even weaker and less consistent effect. While uncertainty ranking often trends slightly downward with increasing depth, the pattern is moderate, and no single depth is uniformly preferred across methods and datasets. We therefore keep depth as a tunable hyperparameter. Activation function. Tanh and ReLU produce near-identical prediction quality. Their effect on uncertainty ranking is modest overall, but tanh appears slightly more robust in cases where ReLU performs poorly. In particular, tanh often avoids some of the weakest epistemic uncertainty rankings observed with ReLU, while rarely introducing a large degradation. The aleatoric results reveal a similar picture: no activation function is clearly better, while tanh is slightly more robust. We therefore adopt tanh as the default activation in the main benchmark as a pragmatic choice: it is competitive with ReLU on average, does not harm prediction quality, and appears safer for epistemic uncertainty ranking in the development experiments. Tuning objective. The tuning-objective ablation reveals a surprisingly weak effect of validation NLL versus validation RMSE. Although one might expect proper-scoring-based tuning to improve uncertainty estimates substantially, the aggregated differences are extremely small and remain well within the variability seen across methods and datasets. In other words, tuning with NLL does not meaningfully improve uncertainty disentanglement in the aggregate, despite its intuitive appeal as the more uncertainty-aware objective. We use validation NLL as a pragmatic default for the large benchmark, but not as evidence that it is uniformly superior to RMSE-based tuning. Optimizer. The optimizer ablation shows no clear overall winner. Average effects on both epistemic and aleatoric ranking are small, so optimizer choice appears to have no significant aggregate impact. We therefore use Adam as a pragmatic default in the main benchmark, since it is lightweight, widely used in deep learning practice, and does not show a consistent disadvantage in the results. A notable qualitative exception concerns methods whose epistemic estimates depend explicitly on the attained minimum, most notably Laplace, FSP-Laplace, and SWAG. For these methods, SOAP tends to be less favorable for epistemic ranking, suggesting that its preconditioning may steer training toward minima whose local geometry is less favorable for epistemic uncertainty estimates. 5.2

Main Benchmark

Predictive performance is similar across models, with only BLR showing a slight disadvantage. We therefore treat the prediction task as sufficiently solved by all models and now discuss the quality of uncertainty disentanglement. Epistemic uncertainty. Figure 2 shows a clear but incomplete separation between methods for epistemic uncertainty. Current methods remain far from near-oracle performance, with most correlations still well below ρ = 0.5, but the methods are not indistinguishable: ensemble-style neural approaches are generally among the strongest, and several non-neural baselines also recover nontrivial epistemic rankings. Additional diagnostics (see Appendix E) support the same picture from a selective-prediction perspective: among neural methods, NN-Ens, NN-Dropout, NN-Bag, and NNFSP provide the most useful epistemic ranking signal, whereas DEUP is comparatively weak; among non-neural baselines, NGBoost-Bag shows the clearest retained-risk reductions, with CatBoost-PS and CatBoost-KGB also remaining useful. Dataset-to-dataset variation is still substantial, even though method rankings are moderately preserved across datasets (W ≈ 0.46), so no method is uniformly reliable. A more diagnostic view comes from the decomposition in Equation (4): the analysis in appendix E.7 shows that most reported epistemic estimates align more closely with the posterior-variance term than with the full posterior excess-risk target, while correlations with the squared-bias term are generally lower. The calibration analysis (see Appendix E.6) is consistent with this interpretation: many methods miscalibrate the scale of epistemic uncertainty, typically 8

Figure 2: Main benchmark results on the held-out benchmark suite. Bars show mean Spearman rank correlation between predicted and oracle epistemic or aleatoric uncertainty. Error bars show ±2 standard errors of the mean across datasets. Grey regions indicate methods that do not provide a heteroscedastic aleatoric estimate. Current methods recover only moderate uncertainty rankings: ensemble-style neural methods, especially NN-Ens and NN-FSP, and tree-based methods such as CatBoost-KGB and NGBoost-Bag are among the stronger epistemic performers, while aleatoric rankings are less clearly separated and remain highly variable across datasets. underestimating it, with EDL being the strongest neural method in calibration. The GP sanity check in Appendix B.3 supports the diagnostic decomposition used for this analysis. Aleatoric uncertainty. Aleatoric uncertainty shows weaker and less stable separation between methods than epistemic uncertainty. In Figure 2, several methods achieve broadly similar average correlations, and the error bars are large, so many apparent pairwise differences are not robust across datasets. This is mirrored by the risk-coverage analysis (see Appendix E.6), where most aleatoric curves lie relatively close to one another and the overall selective signal is weaker than for epistemic uncertainty. Only a few methods stand out modestly, most notably SWAG and EDL among the neural methods and NGBoost-Bag among the non-neural baselines, while several others are close to uninformative. The aleatoric calibration curves are even less favorable: many methods substantially miscalibrate the magnitude of aleatoric uncertainty, often by overestimating it, with EDL strongest among the neural methods and NGBoost-NIG a clear failure case.

6

Limitations

The benchmark relies on an explicitly specified, sampleable posterior over ground-truth functions to compute oracle-uncertainty targets. In the current regression setting, this restricts the exact protocol to Gaussian-process ground truths and makes the benchmark necessarily semi-synthetic. Although the theory is more general, the current oracle computations are instantiated only under squared-error/MSE risk. Extending the benchmark to other strictly proper losses or scoring rules would be important for testing whether the oracle rankings and method comparisons are robust to the chosen risk functional. Extending the same oracle construction to domains such as image classification or image regression is also difficult because exact ground-truth posteriors are typically unavailable. In addition, the current experiments are limited to small-to-moderate tabular datasets. Scaling the oracle protocol to substantially larger datasets remains an open practical challenge.

7

Conclusion

We proposed sample-conditional posterior risk as a direct, estimator-aware target for uncertainty that unifies posterior-variance-based and risk-based views by measuring uncertainty through the expected loss under plausible ground-truth functions. The resulting semi-synthetic evaluation protocol makes oracle aleatoric and epistemic uncertainty computable while retaining realistic covariate structure. Across this setting, current methods often predict accurately but do not reliably disentangle uncertainty, especially under misspecification and sensitivity to modeling and training choices. As a practical 9

takeaway, among general-purpose methods, deep ensembles, FSP-Laplace, and CatBoost-KGB appear to be the most reliable starting points. Still, no method should be used uncritically in high-stakes settings, and direct oracle evaluations are needed to drive progress toward trustworthy uncertainty quantification.

10

Acknowledgments and Disclosure of Funding We acknowledge the financial support of the Gemeinnützige Hertie-Stiftung and the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – 459422098.

References Ben Adlam, Neha Gupta, Zelda Mariet, and Jamie Smith. Understanding the bias-variance tradeoff of Bregman divergences, February 2022. URL http://arxiv.org/abs/2202.04167. arXiv:2202.04167 [stat]. Alexander Amini, Wilko Schwarting, Ava Soleimany, and Daniela Rus. Deep evidential regression, 2020. URL https://arxiv.org/abs/1910.02600. Davide Ballabio, Matteo Cassotti, Viviana Consonni, and Roberto Todeschini. QSAR fish toxicity. UCI Machine Learning Repository, 2015. DOI: https://doi.org/10.24432/C5JG7B. Lukas Biewald. Experiment tracking with weights and biases. https://www.wandb.ai/, 2020. Software available from wandb.ai. Leo Breiman. Bagging predictors. Machine Learning, 24(2):123–140, 1996. doi: 10.1023/A: 1018054314350. Thomas Brooks, D. Pope, and Michael Marcolini. Airfoil Self-Noise. UCI Machine Learning Repository, 1989. DOI: https://doi.org/10.24432/C5VW2C. Christopher Bülte, Yusuf Sale, Timo Löhr, Paul Hofman, Gitta Kutyniok, and Eyke Hüllermeier. An axiomatic assessment of entropy- and variance-based uncertainty quantification in regression, 2025. URL https://arxiv.org/abs/2504.18433. João Caldeira and Björn Nord. Deeply uncertain: Comparing methods of uncertainty quantification in deep learning algorithms. Machine Learning: Science and Technology, 2(1):015002, 2021. doi: 10.1088/2632-2153/aba6f3. Bertrand Charpentier, Ransalu Senanayake, Mykel Kochenderfer, and Stephan Günnemann. Disentangling epistemic and aleatoric uncertainty in reinforcement learning, 2022. URL https: //arxiv.org/abs/2206.01558. Tristan Cinquin, Marvin Pförtner, Vincent Fortuin, Philipp Hennig, and Robert Bamler. Fsp-laplace: Function-space priors for the laplace approximation in bayesian deep learning, 2024. URL https://arxiv.org/abs/2407.13711. Paulo Cortez, António Cerdeira, Fernando Almeida, Telmo Matos, and José Reis. Modeling wine preferences by data mining from physicochemical properties. Decision Support Systems, 47(4): 547–553, 2009. doi: 10.1016/j.dss.2009.05.016. Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace redux - effortless bayesian deep learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 20089–20103. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/file/ a7c9585703d275249f30a088cebba0ad-Paper.pdf. DELVE. Computer Activity Database. http://www.cs.toronto.edu/~delve/copyright. html, n.d. Referenced by the OpenML Croissant metadata for cpu_activity as the dataset citation/source attribution. Tony Duan, Anand Avati, Daisy Yi Ding, Khanh K. Thai, Sanjay Basu, Andrew Ng, and Alejandro Schuler. Ngboost: natural gradient boosting for probabilistic prediction. In Proceedings of the 37th International Conference on Machine Learning, ICML’20. JMLR.org, 2020. 11

Shahriar Faghani, Mana Moassefi, Pouria Rouzrokh, Bardia Khosravi, Francis Baffour, Michael D Ringler, and Bradley J. Erickson. Quantifying uncertainty in deep learning of radiologic images. Radiology, 308(2), aug 2023. doi: 10.1148/radiol.222217. URL https://doi.org/10.1148/ radiol.222217. Alexander Fishkov, Kajetan Schweighofer, Mykyta Ielanskyi, Nikita Kotelevskii, Mohsen Guizani, and Maxim Panov. Uncertainty quantification for regression using proper scoring rules, 2025. URL https://arxiv.org/abs/2509.26610. Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1050–1059, New York, New York, USA, 20–22 Jun 2016. PMLR. URL https://proceedings.mlr.press/v48/gal16.html. Zoubin Ghahramani. Kin Family of Datasets. https://www.cs.toronto.edu/~delve/data/ kin/desc.html, 1996. DELVE dataset family; last updated 5 October 1996. Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 18932–18943. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/ paper_files/paper/2021/file/9d86d83f925f2149e9edb0ac3b49229c-Paper.pdf. Fredrik K. Gustafsson, Martin Danelljan, and Thomas B. Schön. How reliable is your regression model’s uncertainty under real-world distribution shifts?, 2023. URL https://arxiv.org/abs/ 2302.03679. Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning, 2011. URL https://arxiv.org/abs/1112.5745. Ling Huang, Su Ruan, Yucheng Xing, and Mengling Feng. A review of uncertainty quantification in medical image analysis: Probabilistic and non-probabilistic methods. Medical Image Analysis, 97: 103223, 2024. doi: 10.1016/j.media.2024.103223. URL https://doi.org/10.1016/j.media. 2024.103223. Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods. Machine Learning, 110(3):457–506, 2021. doi: 10.1007/ S10994-021-05946-3. URL https://doi.org/10.1007/s10994-021-05946-3. Sebastián Jiménez, Mira Jürgens, and Willem Waegeman. Why machine learning models fail to fully capture epistemic uncertainty, 2025. URL https://arxiv.org/abs/2505.23506. Mira Jürgens, Nis Meinert, Viktor Bengs, Eyke Hüllermeier, and Willem Waegeman. Is epistemic uncertainty faithfully represented by evidential deep learning methods?, 2024. URL https: //arxiv.org/abs/2402.09056. Kaggle. House Sales in King County, USA. https://www.kaggle.com/datasets/harlfoxem/ housesalesprediction, n.d. Original source referenced by OpenML dataset 44989. License reported in OpenML metadata: CC0: Public Domain. Toni Karvonen and François Bachoc. Scale estimation and rate-unbiasedness for gaussian processes under smoothness misspecification, 2025. URL https://arxiv.org/abs/2110.02810. Originally submitted 2021; substantially revised 2025 (v3 on arXiv). Toni Karvonen, George Wynne, Filip Tronarp, and Chris J. Oates. Maximum likelihood estimation and uncertainty quantification for gaussian process approximation of deterministic functions, 2020. URL https://arxiv.org/abs/2001.10965. Yuko Kato, David M. J. Tax, and Marco Loog. A view on model misspecification in uncertainty quantification, 2022. URL https://arxiv.org/abs/2210.16938. Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision?, 2017. URL https://arxiv.org/abs/1703.04977. 12

Andreas Kirsch, Jishnu Mukhoti, Joost van Amersfoort, Philip H. S. Torr, and Yarin Gal. On pitfalls in ood detection: Predictive entropy considered harmful. In ICML 2021 Workshop on Uncertainty and Robustness in Deep Learning, 2021. URL https://www.gatsby.ucl.ac. uk/~balaji/udl2021/accepted-papers/UDL2021-paper-092.pdf. Presented at the ICML 2021 Workshop on Uncertainty and Robustness in Deep Learning. Benjamin Kompa, Jasper Snoek, and Andrew L. Beam. Second opinion needed: communicating uncertainty in medical machine learning. NPJ Digital Medicine, 4:4, January 2021. ISSN 23986352. doi: 10.1038/s41746-020-00367-3. URL https://pmc.ncbi.nlm.nih.gov/articles/ PMC7785732/. Nikita Kotelevskii, Aleksandr Artemenkov, Kirill Fedyanin, Fedor Noskov, Alexander Fishkov, Artem Shelmanov, Artem Vazhentsev, Aleksandr Petiushko, and Maxim Panov. Nonparametric uncertainty quantification for single deterministic neural network, 2022. URL https://arxiv. org/abs/2202.03101. Nikita Kotelevskii, Vladimir Kondratyev, Martin Takáč, Éric Moulines, and Maxim Panov. From risk to uncertainty: Generating predictive uncertainty measures via bayesian estimation, 2024. URL https://arxiv.org/abs/2402.10727. Salem Lahlou, Moksh Jain, Hadi Nekoei, Victor Ion Butoi, Paul Bertin, Jarrid Rector-Brooks, Maksym Korablyov, and Yoshua Bengio. Deup: Direct epistemic uncertainty prediction, 2021. URL https://arxiv.org/abs/2102.08501. Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles, 2017. URL https://arxiv.org/abs/1612. 01474. Yucen Lily Li, Daohan Lu, Polina Kirichenko, Shikai Qiu, Tim G. J. Rudner, C. Bayan Bruss, and Andrew Gordon Wilson. Out-of-distribution detection methods answer the wrong questions, 2025. URL https://arxiv.org/abs/2507.01831. David J. C. MacKay. Bayesian interpolation. Neural Computation, 4(3), May 1992. doi: 10.1162/ neco.1992.4.3.415. URL https://doi.org/10.1162/neco.1992.4.3.415. Wesley Maddox, Timur Garipov, Pavel Izmailov, Dmitry Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning, 2019. URL https://arxiv.org/ abs/1902.02476. Bálint Mucsányi, Michael Kirchhof, and Seong Joon Oh. Benchmarking uncertainty disentanglement: Specialized uncertainties for specialized tasks, 2024. URL https://arxiv.org/abs/2402. 19460. Warwick Nash, Tracy Sellers, Simon Talbot, Andrew Cawthorn, and Wes Ford. Abalone. UCI Machine Learning Repository, 1994. DOI: https://doi.org/10.24432/C55C7W. Craig A. Olson. A comparison of parametric and semiparametric estimates of the effect of spousal health insurance coverage on weekly hours worked by wives. Journal of Applied Econometrics, 13 (5):543–565, 1998. doi: 10.1002/(SICI)1099-1255(1998090)13:5<543::AID-JAE507>3.0.CO;2-J. OpenML. cpu_activity. https://www.openml.org/d/44978, 2022a. OpenML dataset, Data ID 44978, version 6. License: Public. Original source: https://www.dcc.fc.up.pt/~ltorgo/ Regression/DataSets.html. OpenML. diamonds. https://www.openml.org/d/44979, 2022b. OpenML dataset, Data ID 44979, version 8. License: Public. Original source: https://ggplot2.tidyverse.org/ reference/diamonds.html. OpenML. health_insurance. https://www.openml.org/d/44993, 2022c. OpenML dataset, Data ID 44993, version 3. License: GPL (>= 2). Original source: https://rdrr.io/cran/Ecdat/ man/HI.html. 13

OpenML. kin8nm. https://www.openml.org/d/44980, 2022d. OpenML dataset, Data ID 44980, version 4. License: Public. Original source: https://www.cs.toronto.edu/~delve/data/ kin/desc.html. OpenML. kings_county. https://www.openml.org/d/44989, 2022e. OpenML dataset, Data ID 44989, version 2. License: CC0: Public Domain. Original source: https://www.kaggle.com/ datasets/harlfoxem/housesalesprediction. OpenML. physiochemical_protein. https://www.openml.org/d/44963, 2022f. OpenML dataset, Data ID 44963, version 7. License: CC BY 4.0. Original source: https://archive.ics.uci. edu/ml/datasets/Physicochemical+Properties+of+Protein+Tertiary+Structure. OpenML. sarcos. https://www.openml.org/d/44976, 2022g. OpenML dataset, Data ID 44976, version 9. License: Public. Original source: http://gaussianprocess.org/gpml/data/. OpenML. white_wine. https://www.openml.org/d/44971, 2022h. OpenML dataset, Data ID 44971, version 7. License: CC BY 4.0. Original source: https://archive.ics.uci.edu/ml/ datasets/wine. OpenML. space_ga. https://www.openml.org/d/45402, 2023. OpenML dataset, Data ID 45402, version 5. License: Public. Source listed in OpenML metadata: https://www.openml.org/ search?type=data&sort=runs&id=507&status=active. R. Kelley Pace and Ronald Barry. Quick computation of spatial autoregressive estimators. Geographical Analysis, 29(3):232–247, 1997. doi: 10.1111/j.1538-4632.1997.tb00959.x. David Pfau. A generalized bias-variance decomposition for bregman divergences, 2025. URL https://arxiv.org/abs/2511.08789. Janis Postels, Mattia Segù, Tao Sun, Luca Daniel Sieber, Luc Van Gool, Fisher Yu, and Federico Tombari. On the practicality of deterministic epistemic uncertainty. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 17870–17909. PMLR, 2022. URL https://proceedings.mlr. press/v162/postels22a.html. Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features, 2019. URL https://arxiv.org/ abs/1706.09516. Prashant Rana. Physicochemical Properties of Protein Tertiary Structure. UCI Machine Learning Repository, 2013. DOI: https://doi.org/10.24432/C5QW3H. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge, 2015. URL https://arxiv.org/abs/1409.0575. Kajetan Schweighofer, Lukas Aichberger, Mykyta Ielanskyi, Günter Klambauer, and Sepp Hochreiter. Quantification of uncertainty with adversarial models, 2023. URL https://arxiv.org/abs/ 2307.03217. NeurIPS 2023. Lewis Smith and Yarin Gal. Understanding measures of uncertainty for adversarial example detection, 2018. URL https://arxiv.org/abs/1803.08533. Jasper Snoek, Yaniv Ovadia, Emily Fertig, Balaji Lakshminarayanan, Sebastian Nowozin, D. Sculley, Joshua V. Dillon, Jie Ren, and Zachary Nado. Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019 (NeurIPS 2019), pages 13969–13980, 2019. URL https://proceedings. neurips.cc/paper/2019/hash/8558cb408c1d76621371888657d2eb1d-Abstract.html. UCI. Seoul Bike Sharing Demand. https://doi.org/10.24432/C5F62R.

UCI Machine Learning Repository, 2020.

14

DOI:

Aleksei Ustimenko, Artem Beliakov, and Liudmila Prokhorenkova. Gradient boosting performs gaussian process inference, 2023. URL https://arxiv.org/abs/2206.05608. Matias Valdenegro-Toro and Daniel Saromo. A deeper look into aleatoric and epistemic uncertainty disentanglement. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1508–1516, 2022. doi: 10.1109/CVPRW56347.2022.00157. URL https://doi.org/10.1109/CVPRW56347.2022.00157. Vladimir N. Vapnik. The Nature of Statistical Learning Theory. Springer, 2000. ISBN 978-1-44193160-3 978-1-4757-3264-1. doi: 10.1007/978-1-4757-3264-1. URL http://link.springer. com/10.1007/978-1-4757-3264-1. Sethu Vijayakumar and Stefan Schaal. Locally weighted projection regression: An o(n) algorithm for incremental real time learning in high dimensional spaces. In Proceedings of the Seventeenth International Conference on Machine Learning, pages 288–293, 2000. Tobias Weber, Bálint Mucsányi, Lenard Rommel, Thomas Christie, Lars Kasüschke, Marvin Pförtner, and Philipp Hennig. Laplax: Laplace approximations in jax. https://github.com/ laplax-org/laplax, 2025. Hadley Wickham. Prices of over 50,000 round cut diamonds. https://ggplot2.tidyverse.org/ reference/diamonds.html, n.d. Dataset documentation for the diamonds dataset in ggplot2. Original source referenced by OpenML dataset 44979. Lisa Wimmer, Yusuf Sale, Paul Hofman, Bernd Bischl, and Eyke Hüllermeier. Quantifying aleatoric and epistemic uncertainty in machine learning: Are conditional entropy and mutual information appropriate measures?, 2022. URL https://arxiv.org/abs/2209.03302. I-Cheng Yeh. Concrete Compressive Strength. UCI Machine Learning Repository, 1998. DOI: https://doi.org/10.24432/C5PK67.

15

Appendix Contents A Specializations of Eq. (4)

17

A.1 Squared loss (MSE) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

17

A.2 CRPS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

18

A.3 Negative log-likelihood (log score) . . . . . . . . . . . . . . . . . . . . . . . . . .

19

B Diagnostic tools for epistemic uncertainty

20

B.1 Decomposing epistemic uncertainty into estimation and approximation contributions

20

B.2 Approximating the diagnostic targets . . . . . . . . . . . . . . . . . . . . . . . . .

20

B.3 Empirical sanity check . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

21

C Datasets

22

D Models

26

D.1 Probabilistic Regression Baseline . . . . . . . . . . . . . . . . . . . . . . . . . . .

26

D.2 Deep Kernel Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

26

D.3 Tree-Based Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

26

D.4 Bayesian Neural Network Models . . . . . . . . . . . . . . . . . . . . . . . . . .

27

E Additional Results

32

E.1 Depth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

32

E.2 Width . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

32

E.3 Activation Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

33

E.4 Tuning Objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

34

E.5 Optimizers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

36

E.6 Main Benchmark . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

36

E.7 What do epistemic estimates track? Posterior variance, bias, and posterior excess risk 39

16

A

Specializations of Eq. (4)

Fix an input x and the observed dataset S. Recall the sample-conditional posterior risk   RS (fˆ; x) := Ef ∼Π(·|S) EY ∼P (·|x,f ) ℓ(fˆ(x), Y ) .

Throughout this appendix, we assume that all displayed expectations and integrals are finite and that the relevant Bayes acts and posterior centroids exist. We consider both proper pointwise losses, where fˆ(x) is a point prediction, and proper scoring rules, where fˆ(x) is a predictive distribution. Let D ⊆ Rd be a convex set and let ϕ : D → R be strictly convex and continuously differentiable. The Bregman divergence induced by ϕ is Dϕ (u, v) := ϕ(u) − ϕ(v) − ⟨∇ϕ(v), u − v⟩,

u, v ∈ D.

To match the notation of Equation (3) in the main text, we introduce the conditional Bayes act af (x) ∈ arg min EY |x,f [Dϕ (UY ∥ z)] z

and the corresponding posterior Bregman centroid ā(x) ∈ arg min Ef |S [Dϕ (af (x) ∥ z)] . z

With this notation, Equation (3) can be restated as h h  ii Ef EY |x,f Dϕ UY fˆ(x)     = Ef EY |x,f [Dϕ (UY ∥ af (x))] + Ef [Dϕ (af (x) ∥ ā(x))] + Dϕ ā(x) fˆ(x) . {z } | {z } | | {z } generalized variance

Bayes error (AU)

(5)

generalized bias

Thus, for each loss or score, it suffices to identify the Bayes act af (x) and the corresponding centroid ā(x). A.1

Squared loss (MSE)

In this subsection, fˆ(x) ∈ R is a point prediction and ℓ(fˆ(x), Y ) = (Y − fˆ(x))2 . Assume the additive-noise model E[ε | x, f ] = 0,

Y = f (x) + ε,

Var(ε | x, f ) = σε2 (x),

so that the conditional noise variance depends only on x. Take UY := Y and ϕ(u) = u2 . Then Dϕ (u∥v) = (u − v)2 . The conditional Bayes act is   af (x) = arg min EY |x,f (Y − z)2 = E[Y | x, f ] = f (x). z∈R

The posterior centroid is therefore ā(x) = Ef |S [f (x)] =: mS (x).

Substituting into (5) gives     RS (fˆ; x) = Ef EY |x,f (Y − f (x))2 + Ef (f (x) − mS (x))2 + (fˆ(x) − mS (x))2 {z } {z } | | {z } | posterior variance

AU

where

= σε2 (x) + σf2 (x) + (fˆ(x) − mS (x))2 ,

Hence, under squared loss,

RS (fˆ; x) = σε2 (x) + σf2 (x) + (fˆ(x) − mS (x))2 . | {z } | {z } EU

17

bias

(7)

σf2 (x) := Varf ∼Π(·|S) (f (x)).

AU

(6)

A.2

CRPS

In this subsection, fˆ(x) is a predictive CDF. Let Fx denote the predicted CDF at x, and let af (x)(t) := Gf (t) := P (Y ≤ t | x, f ) denote the true conditional CDF under latent function f . Thus, in the CRPS case, the Bayes act af (x) is the conditional CDF itself. Recall the CRPS representation Z Fx (t) − 1{Y ≤ t}

CRPS(Fx , Y ) =

2

dt,

R

and assume all such integrals are finite. For fixed f , expand around the Bayes act Gf : Z
2 EY |x,f [CRPS(Fx , Y )] = EY |x,f Fx (t) − 1{Y ≤ t} dt ZR  2 = EY |x,f Fx (t) − Gf (t) + Gf (t) − 1{Y ≤ t} dt R Z
2 = EY |x,f [CRPS(Gf , Y )] + Fx (t) − Gf (t) dt,

(8) (9) (10)

R

because EY |x,f [1{Y ≤ t}] = Gf (t) makes the cross-term vanish pointwise in t. The posterior centroid is the posterior mean CDF ā(x)(t) = ḠS (t) := Ef |S [Gf (t)] = P (Y ≤ t | x, S).

Averaging over f ∼ Π(· | S) and expanding once more gives RSCRPS (Fx ; x) := Ef EY |x,f [CRPS(Fx , Y )]

(11) Z

= Ef EY |x,f [CRPS(Gf , Y )] + Ef {z } | | AU Z + |R

ḠS (t) − Fx (t) {z generalized bias

2

R

Gf (t) − ḠS (t) {z

generalized variance

dt . }

2

dt }

(12)

(13)

Under the additive model Y = f (x) + ε, with noise CDF Fε (· | x), we have

ε ⊥ f | x,

Gf (t) = Fε (t − f (x) | x). Using the change of variables u = t − f (x) and Y = f (x) + ε, Z
2 CRPS(Gf , Y ) = Gf (t) − 1{Y ≤ t} dt ZR
2 = Fε (t − f (x) | x) − 1{ε ≤ t − f (x)} dt ZR
2 = Fε (u | x) − 1{ε ≤ u} du R

= CRPS(Fε , ε).

(14) (15) (16) (17)

Hence EY |x,f [CRPS(Gf , Y )] = Eε [CRPS(Fε , ε)] , which is independent of f (x). Therefore, under CRPS, the aleatoric term depends only on the noise law at x. 18

A.3

Negative log-likelihood (log score)

In this subsection, fˆ(x) is a predictive density. Assume that, for each latent function f , the conditional law of Y | x, f admits a density af (x)(y) := pf (y) := p(y | x, f ) with respect to a common dominating measure µ. Thus, in the NLL case, the Bayes act af (x) is the conditional density itself. Let qx be a predictive density. Assume Z pf (y) | log pf (y)| dµ(y) < ∞,

Z pf (y) | log qx (y)| dµ(y) < ∞,

and that qx (y) > 0 for p̄S -almost every y, where the posterior centroid is the posterior predictive density ā(x)(y) = p̄S (y) := Ef |S [pf (y)] = p(y | x, S). The sample-conditional posterior risk under NLL is RSNLL (qx ; x) := Ef EY |x,f [− log qx (Y )] Z = Ef pf (y) (− log qx (y)) dµ(y). For each f , add and subtract log pf : Z Z pf (y) dµ(y) EY |x,f [− log qx (Y )] = pf (y) (− log pf (y)) dµ(y) + pf (y) log qx (y) = H(pf ) + KL(pf ∥qx ).

(18) (19)

(20) (21)

Therefore

RSNLL (qx ; x) = Ef [H(pf )] + Ef [KL(pf ∥qx )] . Now use the KL bias–variance identity around the centroid p̄S : Ef [KL(pf ∥qx )] = Ef [KL(pf ∥p̄S )] + KL(p̄S ∥qx ).

(22)

Indeed, Z

  Z pf p̄S Ef [KL(pf ∥qx )] = Ef pf log dµ + Ef dµ pf log p̄S qx Z p̄S = Ef [KL(pf ∥p̄S )] + p̄S log dµ qx = Ef [KL(pf ∥p̄S )] + KL(p̄S ∥qx ).

(23) (24) (25)

Hence RSNLL (qx ; x) = Ef [H(pf )] + Ef [KL(pf ∥p̄S )] + KL(p̄S ∥qx ) . | {z } | {z } | {z } AU

generalized variance

(26)

generalized bias

Under additive Gaussian noise Y = f (x) + ε, we have

ε ∼ N (0, σε2 (x)),

ε ⊥ f | x,

pf (·) = N f (x), σε2 (x) ,

and therefore

1 log 2πe σε2 (x) , 2 which is independent of f (x). Thus, under NLL, the aleatoric term depends only on the local noise variance, while the epistemic term decomposes into the posterior KL variance and the KL bias to the posterior predictive. H(pf ) =

19

B

Diagnostic tools for epistemic uncertainty

B.1

Decomposing epistemic uncertainty into estimation and approximation contributions

This appendix describes an additional diagnostic decomposition of posterior excess risk. The decomposition is not used as a primary benchmark target. Instead, it is used to inspect which part of the epistemic uncertainty is attributable to finite sample estimation and which part is attributable to model-class misspecification. The starting point is the classical decomposition of excess risk into estimation and approximation error [Vapnik, 2000]. In our setting the risk is the sample-conditional posterior risk. The key technical point is that the comparator is a global best-in-class function in the hypothesis class. We then inspect this global comparator pointwise. Consequently, the integrated estimation and approximation terms are non-negative, whereas the induced pointwise estimation contribution can be negative at individual inputs. Let µ denote the input distribution used to evaluate global risk. In the benchmark, µ is the empirical distribution over the evaluation pool. For a predictor f , define the integrated posterior risk R̄S⋆ := EX∼µ [RS⋆ (X)] .

(27)

R̄F := inf R̄S (f ) = R̄S (fF ).

(28)

R̄S (f ) := EX∼µ [RS (f ; X)] ,

Let F be the hypothesis class and define the best-in-class predictor fF ∈ arg min R̄S (f ), f ∈F

f ∈F

The integrated posterior excess risk then decomposes as

R̄S (fˆS ) − R̄S⋆ = R̄S (fˆS ) − R̄S (fF ) + R̄S (fF ) − R̄S⋆ . | {z } | {z } estimation error

(29)

approximation error

Inspecting the same global comparator pointwise yields RS (fˆS ; x) − RS⋆ (x) = RS (fˆS ; x) − RS (fF ; x) + {z } | {z } | epistemic uncertainty at x

pointwise estimation contribution

RS (fF ; x) − RS⋆ (x) | {z }

.

(30)

pointwise approximation contribution

The two pointwise terms in Equation (30) integrate under µ to the non-negative global estimation and approximation errors above. The two terms have the following interpretation: 1. Estimation error. This is the excess posterior risk of the fitted estimator relative to the global best-in-class predictor in F. It captures finite-sample effects, limited coverage, and suboptimal optimization. In the idealized limit of infinite data and perfect optimization, its integrated value should vanish. 2. Approximation error. This is the excess posterior risk of the global best-in-class predictor relative to the posterior Bayes risk. It captures model-class misspecification: even with unlimited data and perfect optimization, the Bayes-optimal posterior predictor may not lie in F. Implications. Although the aleatoric–epistemic split is clean at the level of the posterior risk decomposition, aleatoric noise still affects epistemic uncertainty indirectly because it enters through the observed sample S used to fit fˆS . The global estimation error is non-negative by construction, but the pointwise estimation contribution in Equation (30) need not be. This is because fF is globally optimal under µ, so the fitted estimator fˆS may still be locally better than fF at a particular input x. B.2

Approximating the diagnostic targets

The exact decomposition above requires the best-in-class predictor fF , which is generally unavailable in closed form. We therefore use a Monte Carlo approximation that is only intended as a diagnostic target. 20

Estimation Error

Approximation Error

Normalised error

0.008 0.006 0.004 0.002 0.000 64

128

256

512

1024

64

Number of observations

128

256

512

1024

Number of observations

Linear GP

Well-specified GP

Figure 3: Empirical sanity check for the estimation–approximation decomposition. Normalized estimation and approximation errors as a function of the number of observations. We compare a well-specified GP, which uses the correct data-generating kernel, with an under-capacitated linear GP. Estimation error decreases with more observations, whereas approximation error remains comparatively stable. The linear GP has larger approximation error, consistent with model-class misspecification. We draw posterior functions g1 , . . . , gM ∼ p(f | S). For each sampled function gi , we train the same learning algorithm, with the same hyperparameters as fˆS , on a much larger dataset generated from gi (i) and the benchmark noise model. We denote the resulting reference estimator by fˆF . This reference estimator approximates the best-in-class predictor for the posterior function draw gi . Under squared loss, the pointwise estimation contribution is approximated by M  2  2  1 X ˆ (i) ˆ \ Estim(x) = fS (x) − gi (x) − fF (x) − gi (x) , M i=1

(31)

and the pointwise approximation contribution by M 2 1 X  ˆ(i) \ Approx(x) = fF (x) − gi (x) . M i=1

(32)

This approximation is most reliable for model classes and training procedures where the large-sample reference estimator is close to the population best-in-class solution. It is therefore more direct for convex or analytically tractable estimators, and should be interpreted more cautiously for non-convex models such as neural networks. B.3

Empirical sanity check

As a sanity check, we evaluate whether the diagnostic quantities behave as expected in a controlled setting. We compare a well-specified GP, using the kernel from the data-generating process, with an under-capacitated linear GP. The well-specified GP should mainly suffer from finite-sample estimation error, while the linear GP should exhibit larger approximation error because its hypothesis class cannot represent the nonlinear posterior functions. We average normalized errors (by target variance) over the 7 development datasets. Figure 3 confirms this behavior. As the number of observations increases, the estimation error decreases. This matches the interpretation of estimation error as a finite-sample effect. In contrast, approximation error is stable across sample sizes, because it is determined by the mismatch between the hypothesis class and the posterior function class. The linear GP has substantially larger approximation error than the well-specified GP, while the well-specified GP has lower approximation error but can still exhibit finite-sample estimation error. Thus, the diagnostic decomposition separates the two intended sources of epistemic uncertainty in practice.

21

C

Datasets

Each preset builds a semi-synthetic regression task from a UCI or OpenML tabular dataset. We fit a Gaussian process to the standardized data by maximizing the marginal log likelihood. At benchmark runtime, we instantiate the GP prior with the fitted dataset-specific kernel hyperparameters, draw a latent function at the real covariates to define the synthetic ground-truth targets, and then condition the same prior on the training pairs S = {(xi , yi )}ni=1 . Because the prior used for conditioning is the same prior that generated the target function, the resulting posterior is correctly specified by construction. To control covariate shift, the training inputs are not always drawn uniformly. Instead, the covariates are first standardized, a Mahalanobis distance to the empirical center of the covariate cloud is computed for each point, and the points are partitioned into five quantile bins from most central to most peripheral. The preset then specifies what fraction of observations is drawn from each bin. In the balanced setting these fractions are (0.2, 0.2, 0.2, 0.2, 0.2), so the training sample has approximately the same radial covariate coverage as the full evaluation pool and no intentional covariate shift is introduced. By contrast, the OOD setting oversamples the central bins and undersamples outer bins, while the multimode setting emphasizes inner and outer bins and leaves the middle region sparsely observed. Thus, in the tables, “covariate shift” refers to a deliberate mismatch between the covariate distribution of the observed training sample and that of the full evaluation pool. Table 1: Preset Abalone. Property Source Feature subset d / Target Kernel Parameters Mean function Covariate shift neval Aleatoric std σε Posterior std median σ̂f

Value Abalone (UCI ID 1) Nash et al. [1994] Length, Diameter, Height, Whole weight, Shucked weight, Viscera weight, Shell weight 7 / Rings Arc-cosine (degree 0) σ 2 = 2.001, wvar = 1.000, bvar = 1.000 Zero None — balanced stratified across all Mahalanobis-distance bins 4 177 1.73 0.88

Table 2: Preset Airfoil Self-Noise. Property Source Feature subset d / Target Kernel Parameters Mean function Covariate shift neval Aleatoric std σε Posterior std median σ̂f

Value Airfoil Self-Noise (UCI ID 291) [Brooks et al., 1989] Frequency, attack angle, chord length, free-stream velocity, suction-side displacement thickness 5 / Scaled sound pressure κARD 1/2 + κlin κ1/2 : σ 2 = 0.602, ℓ ∈ [0.69, 13.08]; κlin : σ 2 = 0.407 Constant (−0.361) None — balanced stratified across all Mahalanobis-distance bins 1 503 0.188 0.188

22

Table 3: Preset Concrete Compressive Strength. Property Source Feature subset d / Target Kernel Parameters Mean function Covariate shift neval Aleatoric std σε Posterior std median σ̂f

Value Concrete Compressive Strength (UCI ID 165) Yeh [1998] Cement, blast furnace slag, fly ash, water, superplasticizer, coarse aggregate, fine aggregate, age 8 / Compressive strength κARD RQ + κlin κRQ : σ 2 = 0.346, α = 1.0, ℓ ∈ [0.12, 2.38]; κlin : σ 2 = 0.279 Zero Mild interior-concentrated — 40% from innermost Mahalanobis bin, tapering to 10% in outermost 1 030 3.07 2.33

Table 4: Preset CPU Activity. Property Source Feature subset d / Target Kernel Parameters Mean function Covariate shift neval Aleatoric std σε Posterior std median σ̂f

Value CPU Activity (OpenML ID 44978) [OpenML, 2022a, DELVE, n.d.] All 21 system-activity counters: lread, lwrite, scall, sread, swrite, fork, exec, rchar, wchar, pgout, ppgout, pgfree, pgscan, atch, pgin, ppgin, pflt, vflt, runqsz, freemem, freeswap 21 / usr (CPU time in user mode) ARD κARD 5/2 + κPE κ5/2 : σ 2 = 0.505 with 21 ARD lengthscales; κPE : σ 2 = 0.013, p = 1.0 (exponential), with 21 ARD lengthscales Zero None — balanced stratified across all Mahalanobis-distance bins 5 000 0.533 0.520

Table 5: Preset kin8nm. Property Source Feature subset d / Target Kernel Parameters Mean function Covariate shift neval Aleatoric std σε Posterior std median σ̂f

Value kin8nm (OpenML ID 44980) OpenML [2022d], Ghahramani [1996] θ1 , . . . , θ8 (joint angles) 8 / y (end-effector distance) κARD 3/2 σ 2 = 1.120, ℓ ∈ [2.06, 6.55] Zero OOD interior — 50% from innermost Mahalanobis bin, 0% from outermost 5 000 0.0259 0.0250

Table 6: Preset QSAR Fish Toxicity. Property Source Feature subset d / Target Kernel Parameters Mean function Covariate shift neval Aleatoric std σε Posterior std median σ̂f

Value QSAR Fish Toxicity (UCI ID 504) Ballabio et al. [2015] CIC0, SM1_Dz, GATS1i, NdsCH, NdssC, MLOGP 6 / LC50 κARD 3/2 (near-isotropic) σ 2 = 0.862, ℓ ∈ [0.97, 1.79] Zero Bimodal — 40% from innermost and 40% from outermost Mahalanobis bin; gap in the intermediate region 908 0.457 0.367

23

Table 7: Preset Seoul Bike Sharing Demand. Property Source Feature subset d / Target Kernel Parameters Mean function Covariate shift neval Aleatoric std σε Posterior std median σ̂f

Value Seoul Bike Sharing Demand (UCI ID 560) UCI Hour, temperature, humidity, wind speed, visibility, dew-point temperature, solar radiation, rainfall, snowfall 9 / Rented bike count κARD per + κlin κper : σ 2 = 0.376, T = 1.0, ℓ ∈ [0.036, 2.19]; κlin : σ 2 = 0.223 Constant (0.120) Bimodal — 40% from innermost and 40% from outermost Mahalanobis bin; gap in the intermediate region 5 000 315 217

Table 8: Preset Health Insurance. Property Source Feature subset d / Target Kernel Parameters Mean function Covariate shift neval Aleatoric std σε Posterior std median σ̂f

Value Health Insurance (OpenML ID 44993) OpenML [2022c], Olson [1998] Experience, number of young children, number of older children, husband’s income 4 / Weekly hours worked Arc-cosine (degree 0) σ 2 = 0.576, wvar = 1.000, bvar = 1.000 Zero None — balanced stratified across all Mahalanobis-distance bins 1 500 16.5 3.92

Table 9: Preset King County House Prices. Property Source Feature subset d / Target Kernel Parameters Mean function Covariate shift neval Aleatoric std σε Posterior std median σ̂f

Value King County House Prices (OpenML ID 44989) OpenML [2022e], Kaggle [n.d.] Bedrooms, bathrooms, sqft living/lot/above/basement, floors, waterfront, view, condition, grade, year built/renovated, latitude, longitude, sqft living/lot (nearest 15 neighbours) 17 / Sale price (USD) Matérn 12 (ARD) + Linear κM12 : σ 2 = 0.298, ℓ ∈ [0.66, 17.0]; κlin : σ 2 = 0.072 Zero Moderate — 40% from innermost and 30% from second Mahalanobis bin; taper toward outer region 1 000 10 600 10 600

Table 10: Preset Diamonds. Property Source Feature subset d / Target Kernel Parameters Mean function Covariate shift neval Aleatoric std σε Posterior std median σ̂f

Value Diamonds (OpenML ID 44979) OpenML [2022b], Wickham [n.d.] Carat, depth, table, x, y, z 6 / Price (USD) Radial basis function (ARD) σ 2 = 1.461, ℓ ∈ [0.75, 10.71] Zero None — balanced stratified across all Mahalanobis-distance bins 1 500 1 280 210

24

Table 11: Preset Physicochemical Properties of Protein Tertiary Structure. Property Source Feature subset d / Target Kernel Parameters Mean function Covariate shift neval Aleatoric std σε Posterior std median σ̂f

Value Physicochemical Properties of Protein Tertiary Structure (OpenML ID 44963) OpenML [2022f], Rana [2013] F1–F9 (physicochemical descriptors) 9 / RMSD Rational quadratic (ARD) + Linear κRQ : σ 2 = 0.867, ℓ ∈ [0.081, 1.016], α = 1.0; κlin : σ 2 = 0.257 Zero None — balanced stratified across all Mahalanobis-distance bins 1 500 0.557 0.191

Table 12: Preset Sarcos. Property Source Feature subset d / Target Kernel Parameters Mean function Covariate shift neval Aleatoric std σε Posterior std median σ̂f

Value Sarcos (OpenML ID 44976) OpenML [2022g], Vijayakumar and Schaal [2000] V1–V21 (joint positions, velocities, accelerations of a 7-DOF robot arm) 21 / Joint torque (V22) Matérn 52 (ARD) σ 2 = 0.905, ℓ ∈ [2.57, 12.18] Zero None — balanced stratified across all Mahalanobis-distance bins 1 500 2.32 1.80

Table 13: Preset Space GA. Property Source Feature subset d / Target Kernel Parameters Mean function Covariate shift neval Aleatoric std σε Posterior std median σ̂f

Value Space GA (OpenML ID 45402) OpenML [2023], Pace and Barry [1997] Population, education, houses, income, x-coordinate, y-coordinate 6 / Log proportion of presidential votes Powered exponential, p=1 σ 2 = 1.820, ℓ ∈ [1.66, 4.12], p = 1 Zero Bimodal — 40% from innermost and 40% from outermost Mahalanobis bin; gap in the intermediate region 1 000 0.054 0.048

Table 14: Preset White Wine Quality. Property Source Feature subset d / Target Kernel Parameters Mean function Covariate shift neval Aleatoric std σε Posterior std median σ̂f

Value White Wine Quality (OpenML ID 44971) OpenML [2022h], Cortez et al. [2009] Fixed acidity, volatile acidity, citric acid, residual sugar, chlorides, free/total sulfur dioxide, density, pH, sulphates, alcohol 11 / Quality score Radial basis function (ARD) σ 2 = 0.499, ℓ ∈ [1.60, 3.07] Zero None — balanced stratified across all Mahalanobis-distance bins 1 500 0.649 0.249

25

D

Models

This appendix summarizes the regression models used in the benchmark. All methods expose the same prediction interface: a predictive mean ŷ(x), an aleatoric uncertainty estimate, an epistemic uncertainty estimate, and a total uncertainty estimate. Whenever a method is trained in a standardized target space, predictive means and variances are transformed back to the original target scale before evaluation. Unless noted otherwise, hyperparameters were selected with Bayesian sweeps over the search spaces listed below. For tuning, we used 10 Weights & Biases Bayesian hyperparameter optimization sweeps [Biewald, 2020]. Because the models and datasets are small and many runs are required, GPU execution does not significantly outperform CPU execution on these presets. All experiments were run on an 8 Core Apple M3 chip. Reported runtimes are wall-clock times, not CPU-hours. The full experimental pipeline completed in less than 100 hours of wall-clock time. The ablations ran in between 3-9 hours while the final evaluation took 4hours. D.1

Probabilistic Regression Baseline

Bayesian linear regression. The Bayesian linear regression baseline is a conjugate Gaussian linear model with an optional intercept term. It places an isotropic Gaussian prior on the regression weights, w ∼ N (0, α−1 I), and assumes a homoscedastic Gaussian observation model with fixed noise standard deviation σ, y | x, w ∼ N (x⊤ w, σ 2 ). Given the design matrix X, the posterior over the weights is available in closed form,
−1 SN = αI + βX ⊤ X , mN = βSN X ⊤ y, β = σ −2 , up to a small diagonal jitter for numerical stability. The predictive distribution for a test input x is then Gaussian with mean µ(x) = x⊤ mN and variance 2 σpred (x) = σ 2 + x⊤ SN x. Following the implementation, the quadratic term x⊤ SN x is interpreted as epistemic uncertainty, the fixed noise variance σ 2 as aleatoric uncertainty, and the total predictive uncertainty as their sum [MacKay, 1992]. Hyperparameter search space: α ∈ [10−4 , 10] (log-uniform), σ ∈ [10−2 , 2] (log-uniform), and fit_intercept ∈ {True, False}. D.2

Deep Kernel Learning

Deep kernel learning. The DKL model combines a neural feature extractor with a Gaussianprocess output layer. Inputs are mapped to a learned feature representation ϕθ (x) ∈ Rdϕ , and a Gaussian process with learned kernel hyperparameters is placed on this feature space. The predictive mean is the GP posterior mean at ϕθ (x), while predictive variance is decomposed into an aleatoric component given by the learned observation-noise variance and an epistemic component given by the remaining posterior predictive variance. If target standardization is enabled, all reported means and variances are transformed back to the original target scale. Hyperparameter search space: n_layers ∈ {1, 2, 3}, hidden_dim ∈ {16, 32, 64, 128}, feature_space_dim ∈ [2, 8] (integer), noise_std ∈ [0.05, 1.0] (log-uniform), base_kernel_variance ∈ [0.1, 5.0] (loguniform), base_kernel_lengthscale ∈ [0.1, 5.0] (log-uniform), num_iters ∈ [200, 1000] (integer), warmup_steps ∈ [0, 150] (integer), peak_learning_rate ∈ [5·10−4 , 5·10−2 ] (log-uniform), end_learning_rate= 0, gradient_clip ∈ [0.5, 2.0] (uniform), weight_decay ∈ [10−6 , 10−2 ] (log-uniform), and standardize_inputs=True, standardize_targets=True. D.3

Tree-Based Models

NGBoost with Normal-Inverse-Gamma outputs. This model uses NGBoost [Duan et al., 2020] with a Normal-Inverse-Gamma (NIG) predictive metadistribution for evidential regression. The boosting model predicts the parameters of the NIG distribution directly and is trained with the 26

corresponding NIG score; depending on the configuration, the implementation uses the standard NIG log score [Amini et al., 2020] displayed below for the Evidential Deep Learning model. Additional score hyperparameters, such as evidence and KL regularization strengths, can also be set through the wrapper. At prediction time, the wrapper queries NGBoost’s uncertainty decomposition and returns the predictive mean together with the aleatoric, epistemic, and total predictive variances implied by the learned NIG parameters. Optionally, the target can be standardized before fitting, in which case all reported means and variances are transformed back to the original target scale. Hyperparameter search space: n_estimators ∈ [200, 800] (integer), learning_rate ∈ [0.01, 0.2] (loguniform), max_depth ∈ [2, 6] (integer), minibatch_frac ∈ [0.5, 1.0] (uniform), col_sample ∈ [0.5, 1.0] (uniform), natural_gradient=True, use_svgd=False, evid_strength ∈ [10−4 , 1] (log-uniform), kl_strength ∈ [10−4 , 1] (log-uniform), length_scale ∈ [0.1, 5.0] (log-uniform), and standardize_y=True. NGBoost bagging ensemble. The NGBoost bagging baseline uses an ensemble of NGBoost regressors [Duan et al., 2020] with Gaussian predictive distributions and the bagging metadistribution mode. Each ensemble member predicts a Gaussian location and scale. The benchmark prediction is the average of the member means. Aleatoric uncertainty is the average member predictive variance, epistemic uncertainty is the variance across member means, and total uncertainty is the sum of both terms. The ensemble size and bagging fraction are configurable through n_regressors and bagging_frac, respectively. Optionally, the target can be standardized before fitting, with all reported means and variances rescaled to the original target units. Hyperparameter search space: n_estimators ∈ [200, 800] (integer), learning_rate ∈ [0.005, 0.1] (log-uniform), max_depth ∈ [2, 6] (integer), minibatch_frac ∈ [0.5, 1.0] (uniform), col_sample ∈ [0.5, 1.0] (uniform), natural_gradient∈ {True, False}, n_regressors ∈ [5, 25] (integer), sample_fraction ∈ [0.6, 0.95] (uniform), replace∈ {True, False}, and standardize_y∈ {True, False}. CatBoost posterior sampling. This tree ensemble trains multiple CatBoost regressors with posterior sampling and the RMSEWithUncertainty objective. Each member returns a mean and an uncertainty estimate. The benchmark averages member means for prediction, averages memberprovided variances for aleatoric uncertainty, and uses the variance across member means for epistemic uncertainty. When target standardization is enabled, all means and variances are transformed back to the original target scale. Hyperparameter search space: iterations ∈ [100, 600] (integer), learning_rate ∈ [0.01, 0.3] (log-uniform), depth ∈ [3, 8] (integer), l2_leaf_reg ∈ [0.5, 10] (log-uniform), random_strength ∈ [0, 2] (uniform), bagging_temperature ∈ [0, 2] (uniform), n_regressors ∈ [5, 20] (integer), bagging_frac ∈ [0.6, 1.0] (uniform), and standardize_y∈ {True, False}. CatBoost kernel gradient boosting. The CatBoost KGB baseline uses CatBoost’s sample_gaussian_process routine, which implements the kernel gradient boosting sampling procedure of Ustimenko et al. [2023]. In this view, gradient boosting with symmetric trees is used to approximately sample functions from a Gaussian-process posterior under a prior of the form f ∼ GP(0, σ 2 K + δ 2 I). The wrapper draws multiple sampled tree functions and forms predictions by averaging them. Since the implementation only exposes sampled function values and does not provide a separate observation-noise estimate, epistemic uncertainty is taken to be the variance across sampled predictions, aleatoric uncertainty is set to zero, and total uncertainty is equal to the epistemic term. If target standardization is enabled, predictive means and variances are transformed back to the original target scale. We use the implementation available in the CatBoost library [Prokhorenkova et al., 2019]. Hyperparameter search space: posterior_iterations ∈ [200, 1000] (integer), prior_iterations ∈ [50, 300] (integer), learning_rate ∈ [0.03, 0.3] (log-uniform), depth ∈ [4, 8] (integer), sigma ∈ [0.05, 0.5] (log-uniform), delta ∈ [0.01, 0.5] (log-uniform), random_strength ∈ [0.05, 1.0] (log-uniform), eps ∈ [10−5 , 10−3 ] (log-uniform), n_regressors ∈ [5, 20] (integer), and standardize_y∈ {True, False}. D.4

Bayesian Neural Network Models

Shared neural network setup. All JAX/Flax BNN variants use a shared tabular backbone with configurable activation, normalization, dropout, optimizer, and training schedule. By default, we use a residual tabular MLP inspired by TabResNet [Gorishniy et al., 2021]. Given an input x, the

27

backbone first forms

h(0) = Dropout σ(Norm(Win x + bin )) , and then applies L residual blocks of the form   (ℓ) (ℓ)
u(ℓ) = Dropout σ Norm(W1 h(ℓ) + b1 ) ,   (ℓ) (ℓ) ℓ = 0, . . . , L − 1. h(ℓ+1) = σ Norm(W2 u(ℓ) + b2 ) + h(ℓ) , The final backbone representation is passed to a linear output head. For the default residual backbone, width and depth are controlled by resnet_width and resnet_blocks, respectively. Inputs and targets are standardized during training. If ỹ = (y − my )/sy denotes the standardized target, predictive quantities are transformed back to the original target scale via µ(x) = sy µ̃(x) + my ,

v(x) = s2y ṽ(x),

and analogously for aleatoric, epistemic, and total variances. Gaussian-output methods use one of two shared parameterizations. In the standard parameterization, the network predicts a mean and a raw scale parameter, (µraw (x), ρ(x)),

s(x) = softplus(ρ(x)) + ε,

v(x) = s(x)2 .

When optimized with the Gaussian negative log-likelihood, this gives   1 (y − µ(x))2 ℓGNLL (y, x) = log v(x) + , 2 v(x) up to the additive constant 12 log(2π). Alternatively, the same Gaussian predictive family can be represented in natural coordinates. In that case, the network predicts η2 (x) = −(g(η̃2 (x)) + ε) < 0,

(η1 (x), η̃2 (x)), with g given by either softplus or exp, and µ(x) = −

η1 (x) , 2η2 (x)

v(x) = −

1 . 2η2 (x)

The corresponding natural-parameter loss used in the implementation is   η1 (x)2 1 2 ℓNP (y, x) = − η1 (x)y + η2 (x)y + + log(−2η2 (x)) , 4η2 (x) 2 again up to the additive constant 12 log(2π). Unless noted otherwise, methods without a methodspecific objective can be trained with either the standard Gaussian loss or its equivalent naturalparameter form. For methods with M stochastic forward passes or ensemble members {(µm (x), vm (x))}M m=1 , we report M M 1 X 1 X µ̂(x) = µm (x), v̂ale (x) = vm (x), M m=1 M m=1 v̂epi (x) = Varm (µm (x)), v̂tot (x) = v̂ale (x) + v̂epi (x). Shared hyperparameter search space: for all JAX/Flax BNN variants we tuned n_layers∈ {1, 2, 3}, hidden_dim∈ {32, 64, 128, 256, 512}, resnet_width∈ {32, 64, 128, 256, 512}, resnet_blocks∈ {1, 2, 3, 4}, n_epochs∈ [50, 200] (integer), batch_size∈ {64, 128, 256, 512}, lr∈ [10−4 , 5 · 10−3 ] (log-uniform), weight_decay∈ [10−6 , 10−2 ] (log-uniform), and momentum∈ [0.7, 0.95] (uniform). For the main benchmark, Adam was used as the default optimizer with additional tuning over adam_b2∈ [0.95, 0.999] (uniform) and adam_eps∈ [10−9 , 10−6 ] (loguniform). In optimizer ablations, the optimizer-specific spaces were: SGD with sgd_nesterov∈ {False, True}; Muon with muon_ns_steps∈ {3, 5, 7}, muon_adam_b2∈ [0.95, 0.999] (uniform), muon_preconditioning∈ {frobenius, spectral}, and muon_nesterov∈ {True, False}; SOAP with soap_b2∈ [0.9, 0.99] (uniform), soap_precondition_frequency∈ {1, 5, 10, 20}, and soap_precondition_1d∈ {False, True}. 28

BNN bagging. The bagging BNN uses the shared tabular backbone described above with M independently parameterized members. Each member is initialized separately and trained with the same Gaussian-output objective as above, using either the standard Gaussian loss or the naturalparameter loss together with the variance regularizer. In contrast to the deep ensemble below, each member is trained on a fixed subsample of the training data of size determined by sample_fraction, sampled either with or without replacement according to replace. At prediction time, the predictive mean is the average of the member means, aleatoric uncertainty is the average of the member predictive variances, and epistemic uncertainty is the variance across member means; total uncertainty is their sum [Breiman, 1996]. Additional hyperparameter search space: n_members= 10, sample_fraction∈ [0.6, 0.95] (uniform), and replace∈ {True, False}. Deep ensemble BNN. The deep ensemble BNN uses the shared tabular backbone described above with M independently initialized members of identical architecture. All members are trained on the full training set with separate parameters and optimization trajectories, using the same Gaussianoutput objective as above, namely either the standard Gaussian loss or the natural-parameter loss together with the variance regularizer. At prediction time, the predictive mean is obtained by averaging member means, aleatoric uncertainty by averaging member predictive variances, and epistemic uncertainty by the variance across member means; total uncertainty is the sum of the aleatoric and epistemic terms [Lakshminarayanan et al., 2017]. Additional hyperparameter search space: n_members= 10. MC dropout BNN. The MC-dropout BNN uses a single instance of the shared tabular backbone described above and is trained with dropout and the same Gaussian-output objective as above, again using either the standard Gaussian loss or the natural-parameter loss together with the variance regularizer. At test time, dropout remains active and M stochastic forward passes are drawn by sampling different dropout masks. These stochastic predictions are treated analogously to ensemble members: the predictive mean is the average of the sampled means, aleatoric uncertainty is the average of the sampled predictive variances, and epistemic uncertainty is the variance across sampled means; total uncertainty is their sum [Gal and Ghahramani, 2016]. Additional hyperparameter search space: n_members= 10 and dropout∈ [0.05, 0.35] (uniform). Evidential BNN. The evidential BNN uses the shared tabular backbone described above with a four-dimensional evidential head that predicts the parameters of a Normal-Inverse-Gamma (NIG) distribution [Amini et al., 2020], µ(x),

v(x) > 0,

α(x) > 1,

β(x) > 0,

where positivity constraints are enforced through softplus transformations. In the implementation, the model is trained with the NIG marginal negative log-likelihood together with an evidential regularizer, LEDL (x, y) = LNIG (x, y) + λevi Revi (x, y), where λevi corresponds to evi_reg. The NIG marginal negative log-likelihood is

1 LNIG (x, y) = log π − log v(x) − α(x) log 2β(x)(1 + v(x)) 2  
2 1  + α(x) + log v(x) y − µ(x) + 2β(x)(1 + v(x)) 2 
1 + log Γ α(x) − log Γ α(x) + . 2 and the evidential regularizer is
Revi (x, y) = |y − µ(x)| 2v(x) + α(x) .

(33) (34) (35)

At test time, the predictive mean is given by µ(x). Following the implementation, aleatoric uncertainty is computed as β(x) 2 σale (x) = , α(x) − 1 and epistemic uncertainty as β(x) 2 σepi (x) = , v(x)(α(x) − 1) 29

with total uncertainty given by 2 2 2 σtot (x) = σale (x) + σepi (x).

Additional hyperparameter search space: evi_reg∈ [10−3 , 3 · 10−1 ] (log-uniform). DEUP BNN. The DEUP model [Lahlou et al., 2021] consists of two stages built on the shared backbone described above. First, a mean regressor m(x) is trained on a training split using squarederror loss. A held-out validation split is then used to estimate a single global aleatoric variance, X
2 1 2 σ̂ale = yi − m(xi ) . |V| (xi ,yi )∈V

The second stage trains a separate network on the same validation inputs to predict the log excess squared error beyond this global noise level,  n o 
2 2 ri = log max yi − m(xi ) − σ̂ale , 0 +ε , with the targets clipped to a fixed numerical range for stability. At prediction time, the first network provides the predictive mean. The second network outputs a log-epistemic variance, which is 2 2 exponentiated to obtain σ̂epi (x), while σ̂ale is used as a constant aleatoric variance. The total uncertainty is their sum, 2 2 2 σ̂tot (x) = σ̂ale + σ̂epi (x). As for the shared BNN setup, all reported means and variances are transformed back to the original target scale. Additional hyperparameter search space: the first-stage regressor uses the shared BNN search space above. The second-stage excess regressor was additionally tuned over lr_excess∈ [10−4 , 10−1 ] (log-uniform), weight_decay_excess∈ [10−6 , 10−2 ] (log-uniform), n_epochs_excess∈ [30, 200] (integer), and batch_size_excess∈ {32, 64, 128, 256}. Laplace BNN. The Laplace BNN is a post-hoc method applied to a point-estimate network [Daxberger et al., 2021]. We first train the shared Gaussian-head backbone described above to obtain a MAP solution. Following standard last-layer Laplace practice [Daxberger et al., 2021, Weber et al., 2025], we then approximate a Gaussian posterior only over the final mean-head weights and bias, while keeping the backbone parameters and the variance head fixed at their MAP values. The approximation is constructed with laplax [Weber et al., 2025], and the prior precision is calibrated on a held-out validation split using the library’s calibration routine; in addition, we select the posterior temperature post hoc on the same validation split. At prediction time, we sample mean-head parameters from the resulting Gaussian posterior and propagate them through the fixed backbone. Epistemic uncertainty is the variance across the sampled predictive means, whereas aleatoric uncertainty is given by the fixed MAP variance head; total uncertainty is their sum. Additional hyperparameter search space: n_members= 30. Function-space prior Laplace BNN. The function-space prior Laplace BNN is likewise a post-hoc method built on a point-estimate network, but augments MAP training with a function-space prior regularizer [Cinquin et al., 2024]. During training, function values at sampled context points are regularized toward a Gaussian-process prior, with the default implementation using a Matern- 32 plus-linear kernel. After training, we again keep the backbone and variance head fixed at their MAP values and place a Gaussian posterior only over the final mean-head weights. Unlike the standard Laplace model above, this posterior is constructed through a custom low-rank function-space approximation rather than directly through laplax. The posterior temperature is then selected post hoc on a validation split. At prediction time, sampled mean functions yield epistemic uncertainty through their variance, while the MAP variance head provides the aleatoric uncertainty. Additional hyperparameter search space: n_members= 30. SWAG BNN. The SWAG BNN [Maddox et al., 2019] first trains the shared Gaussian-head backbone described above to a point estimate using the standard training objective. Starting from this solution, it then enters a SWAG collection phase in which training is continued along a single SGD trajectory with constant learning rate and momentum. Checkpoints from this trajectory are collected online by

30

maintaining first and second parameter moments, E[θ] and E[θ2 ]. These moments define a diagonal Gaussian posterior approximation over all network parameters,
2 2 σSWAG = E[θ2 ] − E[θ]2 , θ ∼ N µSWAG , s2 diag(σSWAG ) , where µSWAG = E[θ] and s is an optional sampling scale. At prediction time, several networks are sampled from this diagonal SWAG posterior and evaluated with deterministic forward passes. The predictive mean is the average of the sampled means, aleatoric uncertainty is the average sampled observation variance, and epistemic uncertainty is the variance across sampled means. A held-out calibration split can be used to fit a single global variance scale,
2 P 1 (xi ,yi )∈C yi − µ̂(xi ) |C| P , c= 1 2 (xi ,yi )∈C σ̂tot (xi ) |C| which rescales both aleatoric and epistemic variances at test time. The final total uncertainty is the sum of the rescaled aleatoric and epistemic terms. Additional hyperparameter search space: n_members= 30, n_checkpoints∈ [10, 40] (integer), epoch_per_checkpoint∈ [2, 12] (integer), swag_lr∈ [10−3 , 10−1 ] (log-uniform), swag_momentum∈ {0.0, 0.9, 0.95}, calibration_fraction∈ [0.05, 0.25] (uniform), and grad_clip_norm∈ [0.5, 2.0] (uniform).

31

Spearman ρ Spearman ρ

Abalone 0.5 0.0 −0.5

2

Airfoil

4

6

2

QSAR 0.5 0.0 −0.5

2

Concrete

4

6

2

SeoulBike

4

6 Bagging Deep Ens.

2

4

CPU

6

2

4

6

Kin8nm

4

6

EDL DEUP

2 Laplace FSP-Lap.

4

6 SWAG Dropout

Figure 4: Epistemic uncertainty ranking as a function of network depth , shown separately for each development dataset and BNN method. The curves report Spearman rank correlation between predicted epistemic uncertainty and the oracle epistemic target. The results show substantial method– dataset variability: several curves fluctuate across depths and only a weak average tendency toward lower epistemic ranking at larger depths is visible.

E

Additional Results

E.1

Depth

Increasing depth has little effect on point prediction quality, with mean ρŷ remaining essentially stable across all depths (0.84 ± 0.01). The effect on uncertainty ranking is more visible, but also substantially noisier. Across all valid model–dataset pairs, 62.5% exhibit a negative OLS slope in ρepi and 75% in ρale . The mean paired epistemic–aleatoric score decreases from approximately 0.30 at depth 1 to 0.22 at depth 6, averaged over models and datasets. However, the per-dataset curves in Figures 4 and 5 show that this should not be interpreted as a clean monotone depth law. The effect varies considerably across datasets and methods, with some curves nearly flat, some mildly decreasing, and others fluctuating across depths. The main conclusion is therefore cautious: depth appears to influence uncertainty quality more than predictive quality, but the effect is weak relative to the method–dataset variability. There is no single depth that is uniformly best across methods or uncertainty types. At the same time, the slight negative trend for both epistemic and aleatoric ranking suggests that increasing depth indefinitely is unlikely to be beneficial for uncertainty disentanglement. Decision. We tune depth in the main benchmark rather than fixing it. This keeps the search space flexible while avoiding an overinterpretation of the noisy depth trends. E.2

Width

Increasing network width has no significant effect on prediction quality, with mean ρŷ remaining stable across widths (0.83 ± 0.01). The uncertainty estimates are somewhat more sensitive to width, but again the effect is noisy and differs across datasets and methods. Among the BNN methods and development datasets, 57% of model–dataset pairs exhibit a negative OLS slope in epistemic ranking quality with width, while 91% exhibit a negative slope in aleatoric ranking quality. The epistemic effects are small on average: Deep Ensembles, SWAG, and Bagging show the largest negative slopes (∆ρepi / log2 w = −0.029, −0.023, and −0.019, respectively), Laplace is largely unchanged (−0.008), and FSP-Laplace shows a small positive slope (+0.021). Figures 6 and 7 show that these aggregate numbers should be interpreted as suggestive rather than definitive. Many per-dataset curves are flat or irregular, and the apparent degradation is not uniform across methods. The clearest signal is that very wide networks do not provide a reliable improvement in uncertainty ranking and may slightly degrade it, especially for aleatoric uncertainty. Since predictive performance is stable across widths, there is little evidence that the larger widths are worth the additional risk of noisier or worse uncertainty estimates. 32

Spearman ρ

Abalone

Concrete

CPU

0.5 0.0 −0.5 2

Spearman ρ

Airfoil

4

6

2

QSAR

4

6

2

SeoulBike

4

6

2

4

6

Kin8nm

0.5 0.0 −0.5 2

4

6

2

Bagging Deep Ens.

4

6

EDL DEUP

2 Laplace FSP-Lap.

4

6 SWAG Dropout

Spearman ρ

Spearman ρ

Figure 5: Aleatoric uncertainty ranking as a function of network depth, shown separately for each development dataset and BNN method. The curves report Spearman rank correlation between predicted aleatoric uncertainty and the oracle aleatoric target. As for epistemic uncertainty, the effect of depth is noisy and dataset-dependent. The average trend is mildly negative, but the figure does not support selecting a single universally optimal depth. Abalone 0.5 0.0 −0.5

0.5 0.0 −0.5

16

16

32

32

Airfoil

64 128 256 512 16

32

Concrete

64 128 256 512 16

32

64 128 256 512 16

Width

Width

Width

QSAR

SeoulBike

Kin8nm

64 128 256 512 16

32

Width

64 128 256 512 16

32

Width Bagging Deep Ens.

CPU

32

64 128 256 512

Width

64 128 256 512

Width

EDL DEUP

Laplace FSP-Lap.

SWAG Dropout

Figure 6: Epistemic uncertainty ranking as a function of network width, shown separately for each development dataset and BNN method. The curves report Spearman rank correlation between predicted epistemic uncertainty and the oracle epistemic target. The width effect is noisy and method-dependent: some methods show mild degradation with width, others remain nearly flat, and FSP-Laplace shows a small positive average trend. Decision. To avoid possible degradation of uncertainty estimates without overcommitting to a strong width effect, we restrict width tuning in the main benchmark to the moderate range {16, 32, 64}. E.3

Activation Function

We compare ReLU and tanh activations across all BNN methods and datasets, evaluating prediction quality, epistemic uncertainty ranking, and aleatoric uncertainty ranking. Activation choice has no meaningful effect on prediction quality: the mean difference in Spearman ρŷ across all model–dataset pairs is below 0.01 in absolute value for every model. Epistemic uncertainty. Figure 8 compares ReLU and tanh directly for epistemic uncertainty ranking. Each point corresponds to one model–dataset pair, with ReLU performance on the x-axis and tanh performance on the y-axis. Points above the diagonal indicate that tanh improves over ReLU. The overall pattern does not show a large universal activation effect. Many points lie close to the diagonal, indicating that both activations often lead to similar epistemic rankings. However, tanh appears more robust in several cases where ReLU performs poorly: when ReLU produces low or 33

Spearman ρ Spearman ρ

Abalone

Airfoil

Concrete

CPU

1 0 16

32

64 128 256 512 16

32

64 128 256 512 16

32

64 128 256 512 16

Width

Width

Width

QSAR

SeoulBike

Kin8nm

32

64 128 256 512

Width

1 0 16

32

64 128 256 512 16

32

Width

64 128 256 512 16

32

Width Bagging Deep Ens.

64 128 256 512

Width

EDL DEUP

Laplace FSP-Lap.

SWAG Dropout

Figure 7: Aleatoric uncertainty ranking as a function of network width, shown separately for each development dataset and BNN method. The curves report Spearman rank correlation between predicted aleatoric uncertainty and the oracle aleatoric target. Although individual curves are noisy, the aggregate tendency is more consistently negative than for epistemic uncertainty, suggesting that very wide networks may slightly harm aleatoric uncertainty ranking. even negative epistemic correlations, the corresponding tanh runs are often closer to zero or clearly positive. This suggests that tanh can reduce some of the more severe failure cases, even though it is not uniformly better across all methods and datasets. Aleatoric uncertainty. The aleatoric results show a similar picture. Several model–dataset pairs improve under tanh, especially among cases where ReLU gives weak aleatoric rankings. At the same time, many points remain close to the diagonal, and there are also cases where ReLU performs better. We therefore do not interpret the activation choice as a primary driver of aleatoric uncertainty quality. Rather, the result suggests that tanh is a reasonably safe default that sometimes improves robustness without consistently harming performance. Decision. We adopt tanh as the default activation for all BNN models in the main benchmark. This is a pragmatic choice rather than a strong claim that tanh is always superior: prediction quality is essentially unchanged, and uncertainty ranking is often similar between activations. The main advantage of tanh is that it appears to avoid some ReLU failure cases, especially for epistemic uncertainty, while introducing no measurable predictive cost. E.4

Tuning Objective

The tuning-objective ablation shows no strong overall effect of selecting hyperparameters with validation NLL instead of validation RMSE. Across datasets and methods, the differences are mostly small and mixed, with only a slight and relatively consistent advantage for a few methods, most notably EDL and FSP-Laplace (see the tuning-objective plots in the appendix). This small net effect is also visible in the disentangled averages over the heatmaps: for epistemic uncertainty, validation NLL yields only a mean improvement of about +0.0446 Spearman points relative to the optimizerspecific mean, with validation RMSE showing the same effect in the opposite direction, while for aleatoric uncertainty the corresponding shift is still modest at about +0.0635. The task-diversity analysis reinforces this interpretation. Across tuning conditions, epistemic diversity is very small (1 − W ≈ 0.0549), indicating that epistemic rankings are comparatively robust to the choice between NLL and RMSE tuning, whereas aleatoric diversity is much larger (1 − W ≈ 0.4606), showing that aleatoric rankings are far less stable under the same change. This is somewhat surprising, since one might expect NLL-based tuning to improve aleatoric uncertainty more systematically by directly rewarding probabilistic fit, but Figure 9 does not show consistent dominance over RMSE tuning. Decision. We therefore use validation NLL as a pragmatic default for the large benchmark, while treating this as a standardization choice rather than evidence that NLL tuning is uniformly superior for uncertainty disentanglement.

34

Figure 8: Direct comparison of ReLU and Tanh activations for epistemic uncertainty ranking (left) and aleatoric uncertainty ranking (right). Each point corresponds to one model–dataset pair. The x-axis shows Spearman rank correlation when using ReLU, and the y-axis shows the corresponding correlation when using tanh. Points above the diagonal indicate that tanh improves over ReLU; points below the diagonal indicate the opposite. Most points lie near the diagonal, showing that activation choice is not the dominant factor, but tanh is often more robust in cases where ReLU gives weak uncertainty rankings.

m

ik K

8n

Se ou

lB

Q

+0.04

-0.01

+0.28

+0.04

+0.00

+0.01

-0.32

-0.12

+0.94

-0.31

-0.01

-0.14

Deep Ens.

-0.03

-0.02

-0.15

+0.06

+0.04

-0.01

-0.01

-0.04

+0.01

-0.07

+0.80

-0.30

-0.02

-0.05

EDL

+0.68

-0.02

-0.04

+0.15

+0.53

+0.06

+0.57

+0.37

-0.65

+0.04

+0.07

-0.00

-0.06

-0.31

DEUP

+0.04

n/a

n/a

n/a

n/a

n/a

n/a

n/a

n/a

n/a

n/a

n/a

n/a

n/a

Laplace

+0.00

-0.06

-0.02

+0.24

+0.15

+0.03

-0.05

+0.00

-0.06

-0.22

+0.65

-0.41

+0.06

-0.14

FSP-Lap.

+0.27

-0.03

+0.00

+0.50

+0.48

-0.00

-0.02

+0.31

-0.04

+0.03

+0.71

-0.10

+0.03

-0.16

SWAG

-0.52

+0.01

+0.19

+0.46

+0.41

+0.01

+0.01

-0.08

-0.04

-0.01

+0.85

+0.08

+0.03

+0.07

Dropout

+0.49

+0.06

-0.04

+0.01

-0.08

-0.08

-0.03

+0.08

-0.23

-0.10

-0.09

-0.13

+0.05

+0.04

CatBoost

-0.14

-0.09

-0.19

-0.10

+0.13

-0.02

-0.10

+0.37

+0.21

-0.22

+1.43

+0.20

+0.58

-0.00

CatBoost KGB

+0.07

+0.19

-0.08

-0.10

-0.02

+0.00

-0.10

n/a

n/a

n/a

n/a

n/a

n/a

n/a

Linear

-0.20

+0.04

-0.03

+0.00

-0.21

+0.00

-0.00

n/a

n/a

n/a

n/a

n/a

n/a

n/a

NGBoost NIG

+0.09

+0.07

-0.06

-0.14

-0.10

+0.22

+0.09

-0.45

-0.14

+0.12

-0.17

+0.07

+0.01

+0.12

NGBoost Bag.

+0.39

-0.08

-0.02

+0.00

-0.11

-0.10

-0.08

+0.05

+0.05

+0.04

+0.22

+0.54

+0.38

-0.00

in

CP U

SA R

et e

l

Co

nc r

on e

A

+0.01

ba l A

irf oi

m

ik K

8n

Se ou

lB

Q

-0.07

in

CP U

SA R

et e

l

nc r

Co

on e

A

Bagging

ba l A

irf oi

e

Aleatoric e

Epistemic

1.0

NLL-tuned better

0.0

∆ Spearman ρ

0.5

−0.5

−1.0

RMSE-tuned better

Figure 9: Heatmap of the tuning objective condition. Heatmap of the change in Spearman rank correlation when hyperparameters are selected by validation NLL instead of validation RMSE, shown separately for epistemic and aleatoric uncertainty across methods and datasets. Positive values indicate that NLL-based tuning improves the corresponding uncertainty ranking, while negative values indicate an advantage for RMSE-based tuning. The main takeaway is that the effect of the tuning objective is surprisingly weak and highly dataset- and method-dependent: improvements and degradations are both present, but no clear global winner emerges, so NLL tuning should be viewed as a pragmatic default rather than as a uniformly superior uncertainty-aware choice. 35

E.5

Optimizers

The optimizer ablation likewise shows no clear overall winner. Across methods and datasets, the appendix plots reveal a mixed picture in which optimizer effects are present but generally modest, and their direction depends strongly on the specific method–dataset combination rather than following a single uniform pattern. This is also reflected in the aggregated heatmap averages, which remain small throughout. For epistemic uncertainty, the mean signed shifts are +0.0146 for SGD, +0.0050 for Adam, +0.0229 for Muon, and −0.0431 for SOAP. For aleatoric uncertainty, the corresponding values are +0.0150 for SGD, +0.0450 for Adam, −0.0257 for Muon, and −0.0333 for SOAP. However, these averages should be interpreted with caution: their standard deviations are much larger than the mean effects themselves (≈ 0.11 to 0.16 across all optimizers and both targets), indicating substantial variability across method–dataset combinations and making the aggregate differences statistically unconvincing on their own. We therefore do not interpret the small mean advantages of Muon for epistemic ranking or Adam for aleatoric ranking as evidence of a reliable optimizer preference. The task-diversity analysis leads to a similar conclusion. Across datasets, the diversity is much larger than across optimizer conditions, with 1 − W ≈ 0.486 versus 0.110 for epistemic uncertainty and 1 − W ≈ 0.837 versus 0.469 for aleatoric uncertainty. In other words, variation across benchmark tasks exceeds variation induced by optimizer choice itself. At the same time, the heatmaps suggest one more specific qualitative pattern. Methods whose epistemic estimates rely strongly on the particular minimum found by optimization, most notably Laplace, FSP-Laplace, and SWAG, are often hurt by SOAP on epistemic ranking in several datasets. A plausible interpretation is that SOAP’s preconditioning changes the geometry of the solutions reached during training in a way that is especially unfavorable for uncertainty mechanisms built around the attained minimum or its local curvature. We treat this as a method-specific hypothesis rather than a definitive causal claim, but it is the clearest structured optimizer effect visible beyond the noisy aggregate averages. Decision. For the main benchmark, we therefore do not treat the optimizer as a major source of performance differences. Instead, we adopt Adam as a pragmatic default, since it is lightweight, standard in most deep learning pipelines, and does not show a clear systematic downside in Figure 10. E.6

Main Benchmark

36

m

ke

K in

-0.10

-0.09

+0.13

+0.09

-0.13

-0.07

-0.12

-0.00

-0.03

+0.04

+0.08

+0.31

-0.26

-0.02

-0.09 -0.11

EDL

-0.06

+0.22

-0.11

+0.17

+0.51

+0.11

-0.31

+0.29

+0.10

-0.18

+0.28

+0.19

+0.09

DEUP

-0.17

-0.01

-0.06

+0.02

-0.08

+0.05

-0.02

n/a

n/a

n/a

n/a

n/a

n/a

n/a

Laplace

+0.11

+0.06

-0.04

+0.23

+0.36

+0.01

+0.01

-0.01

-0.13

-0.12

+0.25

-0.06

+0.02

-0.05

FSP-Lap.

+0.02

-0.03

-0.10

+0.07

+0.30

-0.03

+0.01

+0.01

+0.00

-0.03

+0.24

-0.08

+0.05

-0.06

SWAG

+0.07

+0.04

-0.07

+0.14

-0.09

+0.00

+0.00

+0.00

+0.12

-0.18

+0.52

-0.10

-0.01

+0.03

Dropout

+0.15

+0.01

+0.01

-0.11

+0.02

+0.03

+0.04

+0.11

+0.05

+0.00

-0.09

-0.08

-0.06

+0.01

Bagging

-0.39

-0.01

+0.06

+0.01

+0.09

-0.01

+0.06

+0.18

+0.16

-0.13

+0.21

+0.13

+0.08

+0.01

Deep Ens.

-0.08

+0.02

+0.04

+0.09

+0.15

-0.05

-0.05

+0.16

-0.01

+0.04

+0.41

+0.08

+0.04

+0.01

EDL

n/a

+0.02

-0.11

-0.12

+0.28

-0.07

-0.51

n/a

+0.19

+0.04

-0.02

-0.26

-0.05

+0.11

DEUP

-0.10

-0.05

+0.06

+0.02

+0.02

-0.11

-0.04

n/a

n/a

n/a

n/a

n/a

n/a

n/a

Laplace

+0.08

+0.15

+0.03

+0.25

-0.27

-0.01

+0.01

+0.05

-0.13

-0.07

+0.27

+0.02

+0.03

-0.05

FSP-Lap.

+0.02

-0.05

+0.04

+0.15

+0.12

-0.01

+0.02

+0.03

+0.02

-0.02

+0.32

+0.16

+0.12

-0.02

SWAG

+0.17

-0.02

+0.07

+0.11

+0.08

+0.03

+0.07

-0.14

-0.22

+0.19

+0.05

+0.01

+0.00

-0.10

Dropout

-0.02

-0.04

+0.01

+0.06

-0.02

+0.01

+0.01

+0.02

-0.10

+0.16

-0.08

+0.14

+0.04

+0.04

0.6

0.4

0.2

SOAP

Muon

0.0 Bagging

+0.08

+0.01

+0.08

-0.01

-0.06

+0.04

-0.00

-0.02

-0.05

+0.09

-0.10

-0.07

+0.04

Deep Ens.

+0.10

+0.08

+0.02

-0.02

+0.05

+0.01

+0.05

-0.01

-0.03

-0.19

-0.50

+0.17

-0.01

-0.03

EDL

+0.07

-0.03

+0.04

-0.05

-0.25

-0.08

+0.62

-0.20

-0.47

+0.01

+0.25

+0.19

+0.04

+0.02

+0.04

DEUP

+0.10

+0.06

n/a

-0.04

+0.03

n/a

+0.06

n/a

n/a

n/a

n/a

n/a

n/a

n/a

Laplace

+0.02

+0.06

+0.00

-0.02

+0.16

+0.01

-0.04

+0.01

-0.09

+0.14

-0.12

+0.13

-0.07

+0.04

FSP-Lap.

+0.03

+0.03

-0.03

+0.11

-0.08

+0.02

+0.06

+0.02

+0.01

-0.15

+0.01

-0.03

-0.07

-0.01

SWAG

-0.10

+0.01

+0.08

-0.15

+0.29

+0.01

-0.01

+0.03

+0.04

+0.06

-0.34

+0.22

+0.05

+0.00

Dropout

-0.10

-0.02

-0.02

-0.00

+0.05

-0.05

-0.06

-0.11

+0.09

-0.18

+0.04

-0.15

+0.05

-0.04

Bagging

+0.16

+0.03

+0.03

-0.01

-0.10

+0.02

-0.01

-0.05

-0.02

-0.09

-0.20

+0.07

-0.05

+0.07

Deep Ens.

-0.12

+0.08

+0.11

+0.15

-0.08

+0.06

-0.00

-0.12

+0.00

+0.07

-0.22

+0.01

-0.02

+0.10

EDL

-0.01

-0.21

+0.18

+0.00

-0.53

+0.04

+0.20

-0.09

+0.19

+0.14

-0.51

-0.12

-0.08

-0.02

DEUP

+0.17

+0.00

n/a

-0.01

+0.03

+0.06

n/a

n/a

n/a

n/a

n/a

n/a

n/a

n/a

Laplace

-0.21

-0.27

+0.00

-0.46

-0.25

-0.01

+0.02

-0.04

+0.35

+0.05

-0.40

-0.09

+0.03

+0.07

FSP-Lap.

-0.07

+0.04

+0.09

-0.33

-0.34

+0.02

-0.09

-0.05

-0.03

+0.20

-0.58

-0.06

-0.10

+0.09

SWAG

-0.14

-0.03

-0.09

-0.10

-0.27

-0.04

-0.06

+0.10

+0.06

-0.06

-0.23

-0.13

-0.05

+0.06

Dropout

-0.03

+0.05

+0.00

+0.05

-0.05

+0.00

+0.01

-0.01

-0.04

+0.02

+0.12

+0.09

-0.03

-0.02

above optimizer mean

∆ Spearman ρ vs optimizer mean

lo

8n

Se o

-0.04

-0.02

ul

Q SA R

Bi

e

-0.06

-0.12

U CP

et Co

+0.07

-0.21

nc r

A irf

oi l

ne A ba

+0.00

-0.17

8n K in

m

ke Se o

-0.17

-0.18

ul

Q SA R

Bi

e et

-0.02

+0.10

U CP

+0.14

nc r Co

oi l A irf

Bagging Deep Ens.

lo

SGD ADAM

Aleatoric

A ba

ne

Epistemic

−0.2

−0.4

−0.6

below optimizer mean

Figure 10: Heatmap of the optimizer condition. Heatmap of the change in Spearman rank correlation relative to the optimizer-wise mean, shown separately for epistemic and aleatoric uncertainty across methods, datasets, and optimizers. Positive values indicate that a given method–dataset pair performs above the corresponding optimizer mean, while negative values indicate below-average performance. The main takeaway is that optimizer effects are heterogeneous and generally modest, with no optimizer dominating across methods and datasets; the clearest structured exception is that SOAP is often less favorable for epistemic ranking in methods whose uncertainty estimates depend strongly on the attained minimum, such as Laplace, FSP-Laplace, and SWAG.

37

Figure 11: Risk-coverage curves for disentangled uncertainties. Risk-coverage curves for predicted aleatoric uncertainty (top), epistemic uncertainty (middle), and total uncertainty (bottom) on the held-out benchmark suite. For each model, test points are sorted from lowest to highest predicted uncertainty, and the retained-set risk is plotted against the fraction of retained points (coverage); lower curves indicate better selective ranking. Risks are normalized by the corresponding full-run risk and averaged across datasets, with NN and non-NN models shown separately. Total and epistemic uncertainty show clearer selective-ranking behavior than aleatoric uncertainty. By contrast, the aleatoric curves are weaker and more clustered, indicating only limited method separation apart from a few modest outliers.

38

Uncertainty Calibration plots 101

NN models

100

Mean oracle / run mean oracle

non-NN models

101

Total

100 100

101

100

101

Epistemic 100

100

10 1

10 1

10 2

10 2

10 1

10 2

100

10 2

10 1

100

101

100

Aleatoric 102

102

101

101

100

100 100

101

Bagging Deep Ens. EDL DEUP

102

Laplace FSP-Lap. SWAG Dropout

Mean predicted / run mean oracle Linear NGBoost NIG NGBoost Bag.

102

CatBoost CatBoost KGB DKL

Figure 12: Uncertainty calibration plots for total uncertainty (top), epistemic uncertainty (middle), and aleatoric uncertainty (bottom), with NN methods on the left and non-NN methods on the right. For each method, predictions are binned by estimated uncertainty and the mean predicted uncertainty in each bin is plotted against the corresponding mean oracle uncertainty, normalized by the run-wise oracle mean; the dashed diagonal marks perfect calibration. The plots show that total uncertainty is typically closer to calibrated scale than the disentangled components, while epistemic and aleatoric uncertainties exhibit larger and more method-dependent deviations from the oracle. E.7

What do epistemic estimates track? Posterior variance, bias, and posterior excess risk

The posterior-risk definition of epistemic uncertainty decomposes, under squared loss, into a posterior variance term and an estimator-dependent squared bias term,
2 EUS (fˆ; x) = σf2 (x) + fˆ(x) − mS (x) . This allows us to ask whether methods that report epistemic uncertainty are actually tracking the full posterior excess-risk target, or whether they mostly recover only one of its subcomponents. Figure 13 compares each method’s epistemic uncertainty estimate against three oracle quantities: the full posterior excess-risk epistemic target, the posterior variance term, and the squared bias term. The main pattern is that most methods are closer to the posterior variance than to the full posterior excess risk. This suggests that current epistemic estimators primarily behave like posterior-spread estimators: they identify regions where plausible ground-truth functions remain variable under the posterior, but only partially account for whether the predictor is accurate for the posterior mean. The squared-bias correlations are generally smaller and often close to uninformative. Thus, when a model’s predictive mean is not close to the posterior mean, most methods show only limited awareness of this estimator-dependent contribution to epistemic uncertainty. This is important because the posterior-risk oracle treats such bias as epistemic: it is reducible error caused by the particular learned predictor, for example through misspecification, regularization, or optimization. Overall, the figure supports the interpretation that present methods capture the Bayesian spread component of epistemic uncertainty more reliably than the estimator-dependent bias component.

39

Spearman ρ

1.00 0.75 0.50 0.25

P EU -D

N

-S N N

N

W AG

SP -F N

-L ap N N

N

p ro -D

-E

ag

ns N N

N N

N

D

N

-E N N

-B

L

L K D

Ca

tB oo

st-

K

st-

G

B

PS

ag

oo

t-B

tB Ca

os Bo G

N

N

G

Bo o

st-

N

BL

IG

R

0.00

Epistemic

Post. Variance

Squared Bias

±2 SEM

Figure 13: Epistemic subpart analysis. Spearman rank correlations between each method’s epistemic uncertainty estimate and three oracle quantities on the validation suite: the full posterior excess-risk epistemic target, the posterior variance term, and the squared bias term. Error bars show ±2 standard errors of the mean across datasets. Most methods correlate more strongly with posterior variance than with the full epistemic target, while correlations with the squared bias term are substantially weaker. This indicates that current methods mostly capture posterior spread, but have limited sensitivity to the estimator-dependent bias contribution that appears when the learned predictor deviates from the posterior mean.

40

Related documents

• In defence of normic de minimis expected utility theory#97931
• Modeling information acquisition: divergences, duality, and discrete choice#635654
• Modeling information acquisition: divergences, duality, and discrete choice#633305
• How to expect a surprising exam#744828
• Luck: An Introduction#739005