arXiv CSopen access

Decision-Aligned Evaluation of Uncertainty Quantification

arXiv:2606.26990v1 [cs.LG] 25 Jun 2026

Decision-Aligned Evaluation of Uncertainty Quantification

Annika Schneider1,2,3,4 Tommy Rochussen1,2,3 Joshua Stiller3,5 Vincent Fortuin2,3,4,6 1 Technical University of Munich 2 Helmholtz AI 3 MCML 4 Konrad Zuse School of Excellence in Reliable AI 5 LMU Munich 6 University of Technology Nuremberg

Abstract Uncertainty estimates in machine learning are typically evaluated using generic metrics such as the negative log-likelihood and expected calibration error, yet good performance on such metrics does not necessarily imply high utility in downstream decisions. We introduce decision-alignment, a criterion that reveals which evaluation metrics meaningfully align with downstream utilities. Applying this framework, we show that many widely used uncertainty metrics are either misaligned with common decision problems or encode pathological prior beliefs about the downstream task. We then propose prior-weighted utility metrics, a special class of proper scoring rules that provides decision-aligned uncertainty evaluation. Across benchmark experiments and real-world case studies, our metrics consistently align with realized decision utility, while conventional metrics do not. Our results surface flaws in the current UQ evaluation protocol and offer a principled extension of existing metrics toward decision-relevant UQ evaluation.

1

Introduction

Uncertainty quantification (UQ) in machine learning (ML) is key for reliable decision-making. Probabilistic ML research aims to develop high-quality UQ methods, facilitating ML usage in safety-critical domains. How UQ quality is measured directly shapes which methods are developed, published, and deployed. Since the “true uncertainty” cannot be observed, UQ evaluation relies on surrogate metrics such as negative log-likelihood (NLL) or expected calibration error (ECE) [1, 2, 3]. However, good performance on such metrics does not necessarily imply high utility in downstream decisions [4, 5, 6]. We argue that UQ evaluation should reflect downstream decision utility directly. In this paper, we take UQ evaluation as our object of study. We focus on general-purpose evaluation where no specific downstream task is fixed. Benchmarks guide progress in ML [7], so by restricting ourselves to generic measures such as the NLL and ECE at this stage, we risk leaving behind methods that do not excel on such metrics but are actually more useful in downstream applications. In this work, we aim to improve the understanding of common UQ evaluation metrics and align UQ evaluation with decision-making. We make the following contributions: 1. We introduce decision-alignment as a formal criterion to show that common UQ evaluation metrics either encode pathological priors or no coherent decision belief at all (Section 3). 2. Based on these findings, we propose prior-weighted utility metrics that capture a model’s value for downstream utilities (Section 4). 3. We demonstrate in classification and regression experiments that only prior-weighted utility metrics reliably align with real downstream utilities (Section 5).1 1 Our code is available at https://github.com/fortuinlab/prior-weighted-utilities.

Preprint.

Table 1: Summary of the main results from Section 3 for classification (left) and regression (right). If the metric is decision-aligned, we state the implicit prior π. Else, we write not aligned. All UQ metrics we consider are either not decision-aligned or encode pathological prior beliefs.

2

Metric

Binary decision

NLL BS

1 π(c) = c(1−c)

Acc

π(c) = 2δ0.5 (c)

ECE MCE

not aligned

not aligned

ECE MCE

not aligned

not aligned

R-AUC E-Det

not aligned

not aligned

R-AUC E-Det

not aligned

not aligned

π(c) = 2

Top-k select.

Metric

Selective pred.

Top-k select.

−2

not aligned

NLL

π(k) = δn (k),

MSE

not aligned for k < n

π(λ) = ελ

for λ, σ 2 > ε > 0

π(λ) = δ∞ (λ), not aligned for λ < ∞

not aligned not aligned

Related work

Here, we summarize the prior work most relevant to our contribution. In Appendix A, we comment in more detail on the scope of our work. Further studies that share similarities with our framework or goals are acknowledged in Appendix B. Common UQ evaluation metrics in ML Current UQ studies in ML primarily evaluate uncertainty through predictive accuracy and calibration metrics. Benchmarking studies such as those of Ovadia et al. [1], Nado et al. [8], and Basora et al. [3] typically assess models using NLL, Brier score (BS), ECE, and ranking-scores such as the area under the receiver operating characteristic curve (AUROC). Among these metrics, proper scoring rules (PSRs) such as the NLL and BS are often considered favorable as they are minimized at the ground-truth predictive distribution, and hence encourage honest uncertainty estimates [9, 10]. In practice, however, the true distribution is unknown, such that we never know whether a model ever truly minimizes the score [11], limiting the value of properness in UQ benchmarking. While the benchmarking tools URSABench [2] and Neural Testbed [12] do incorporate decision-making and bandit-style evaluations respectively, these decision scenarios are artificial and simplified, limiting their transferability to the real-world. In contrast, our prior-weighted utility metrics are aligned with realistic downstream decisions in a principled manner. Connection of PSRs & decisions There is a well-known connection between PSRs and decisionmaking—every decision problem induces a PSR [13]. Further, in binary classification, any PSR can be expressed as a cost-weighted average over dichotomous decisions (under mild regularity conditions, see [14, 15, 16, 9, 10, 17]); a result that inspires our decision-alignment framework. Some regression PSRs can also be expressed in terms of decisions, e.g., the continuous ranked probability score and BS can be expressed as averages over thresholded decisions [9], but this relationship is less general than in classification [18]. These relationships give rise to the assumption that good performance on common PSRs relates to good performance in downstream decisions, but we refute this assumption through our decision-alignment criterion. Connection between calibration & decisions It is unclear whether better performance on calibration metrics leads to better decision-making. While some studies observed that good calibration implies good decisions [19, 20], others argue that metrics such as the ECE are flawed in terms of evaluating the usefulness of a model in downstream decisions [21, 22, 6]. There has been substantial recent work on more decision-centric notions of calibration which aim to provide approximate guarantees (up to some tolerance ε) for downstream decision utilities or restricted families of tasks [23, 24, 25, 26]. These approaches are primarily motivated by training or post-hoc adjustment, and have limited use for benchmarking since ε-guarantees do not allow for consistent ranking of two predictors—differences in downstream utility may lie within the approximation error. In contrast, our prior-weighted utilities are intentionally designed to provide consistent model rankings in terms of downstream utility and are therefore preferable in the context of benchmarking. 2

3

Implicit decision beliefs of UQ metrics

Standard UQ metrics are often justified axiomatically (e.g., through properness or calibration), but there is only a limited understanding of their implications for downstream utilities. We generalize the known connection between PSRs and decisions (see Section 2) to decision-alignment, allowing a systematic analysis of common UQ metrics in terms of decision utilities. 3.1

Decision-alignment of UQ metrics

We consider metrics M and decision utilities Uθ , parametrized by θ ∈ Θ, that evaluate a predictionlabel pair (f , y), where y = (y1 , . . . , yn ) ∈ Y n and f = (f1 , . . . , fn ) ∈ P n , n ≥ 2, and Y, P depend on the task and prediction model. We consider probabilistic predictions, so each f ∈ P encodes a distributional report over Y. For example, in probabilistic binary classification, Y = {0, 1} and P = [0, 1], that is, the prediction model maps to the probability that a given instance has label 1. We restrict ourselves to prediction spaces P and utilities Uθ such that for each f ∈ P a (not necessarily unique) Bayes act exists for Uθ , that is, a decision that maximizes expected utility under f . Without loss of generality, we say that smaller values of M and larger values of U correspond to better performance. Since the labels y are typically fixed, we use the notation2 My (f ) := M (f , y)

and

Uθ,y (f ) := Uθ (f , y).

Definition 3.1 (Decision-alignment). A metric M is decision-aligned w.r.t. a decision family {Uθ }θ∈Θ if there exists a prior π s.t. for any fixed labels y, there exists a strictly increasing function hy : Im(My ) → R s.t. Z
hy My (f ) = −Uθ,y (f ) π(θ) dθ ∀f . (1) Θ

Remark 3.2 (On the nature of π). In our framework, π can be any nonnegative and nonzero measure on Θ. It does not need to be finite and, in particular, does not need to be a probability measure. We only require that the right-hand side of (1) is well-defined and finite. To simplify notation, we write π(θ)dθ for integration w.r.t. the measure π, including absolutely continuous priors with density π(θ), as well as purely atomic priors such as Dirac measures δ. We use the term prior to highlight that π encodes the metric’s implicit belief on the downstream utility’s parameter, and one can think of the right-hand side in (1) as the (negative) expected utility under θ ∼ π. Thus, we introduce the notation Z EUθ∼π,y (f ) := −Uθ,y (f ) π(θ) dθ. Θ

Remark 3.3 (On the choice of our framework). The reason why we defined our framework of the form (1) is two-fold: (1) Metric-analysis: Expressing a metric M through (1) reveals whether it implicitly anticipates some decision family {Uθ }θ∈Θ under a prior π. This helps us decide when and whether to use the metric, and to interpret model rankings under M appropriately. (2) Desirable metric property: Decision-alignment is a necessary and sufficient condition for strict order- and tie-preservation (see Proposition 3.4), and thus a naturally desirable metric property. Intuitively, a metric M is decision-aligned if and only if it ranks predictions exactly as expected utility under π would. Proposition 3.4 (Decision-alignment ⇔ strict order- and tie-preservation). Fix a decision family {Uθ }θ∈Θ and a prior π. Then, for any fixed y, the following statements are equivalent: (I) M is decision-aligned w.r.t. {Uθ }θ∈Θ under π. (II) M is strictly order-preserving and tie-preserving for {Uθ }θ∈Θ under π, i.e., for all f 1 , f 2 , My (f 1 ) < My (f 2 ) ⇒ EUθ∼π,y (f 1 ) < EUθ∼π,y (f 2 ), My (f 1 ) = My (f 2 ) ⇒ EUθ∼π,y (f 1 ) = EUθ∼π,y (f 2 ). Proof sketch. (I)⇒(II): Follows through strict monotonicity of hy . (II)⇒(I): Follows by defining hy (m) as EUθ∼π,y (f ) for any f with My (f ) = m. We state the full proof in Appendix C.1. 2 Note that often, utility functions evaluate actions given labels, and we evaluate predictions given labels. To be precise, we

define Uθ,y (f ) as the utility of the Bayes act under f .

3

In other words, decision-aligned metrics rank models according to their expected utility in the respective downstream decision family. Further, any metric that is strictly order- and tie-preserving is also decision-aligned, so Definition 3.1 includes all metrics that fulfill these criteria. In Appendix D, we discuss the relationship between decision-aligned metrics and PSRs. To understand what kind of metrics can be decision-aligned with respect to what types of utility families, we distinguish between pointwise-separable, coordinate-independent, and coordinatedependent metrics/utilities. Definition 3.5 (Pointwise-separability & coordinate-independence). A metric/utility S is pointwiseseparable if there exists a measurable instance function s s.t.3 n

S(f , y) =

1X s(fi , yi ) n i=1

(2)

Otherwise, S is non-separable. S is coordinate-independent if the change in S through altering one prediction fi does not depend on the other coordinates, that is, for a, b ∈ P and r, r ′ ∈ P n−1 Sy ((a, r)) < Sy ((b, r)) ⇔ Sy ((a, r ′ )) < Sy ((b, r ′ )), and analogously for equality and opposite strict inequality.4 Otherwise, S is coordinate-dependent. Note that each pointwise-separable metric/utility is also coordinate-independent: For a pointwiseseparable S, changing fi only affects s(fi , yi ), which is independent of the remaining predictions. Next, we show that coordinate-dependent metrics (e.g., calibration or ranking metrics) can immediately be ruled out as decision-aligned with respect to pointwise-separable utility families. Lemma 3.6 (Separability barrier). If the utility Uθ is pointwise-separable for every θ ∈ Θ and θ 7→ Uθ (f , y) is integrable for all (f , y), then any metric M that is decision-aligned w.r.t. {Uθ }θ∈Θ (for some prior π) must be coordinate-independent. Proof sketch. By
exchanging the integral and pointwise sum on the right-hand side of (1), we get that hy My (f ) is pointwise-separable, which requires M to be coordinate-independent. We state the full proof in Appendix C.2. Note that the converse of Lemma 3.6 is not true: A coordinate-independent metric M can be decisionaligned w.r.t. {Uθ }θ∈Θ even if every Uθ is non-separable, as long as the non-separable parts cancel out in the integral (see Appendix E for an example). We now analyze concrete examples of metric-decision pairings through the lens of decision-alignment. We consider decision tasks that arise naturally in applications and span both pointwise-separable and non-separable utilities—binary decisions (yes/no), selective prediction (only predict when sure enough), and top-k selection (select a few items from a large candidate pool). 3.2

Classification

In this section, we consider probabilistic binary5 classification models f : X → [0, 1], mapping from the input space X to the probability that the true label from the output space Y = {0, 1} is 1. As evaluation metrics, we consider the negative log-likelihood (NLL), Brier score (BS), accuracy (Acc), expected calibration error (ECE), maximum calibration error (MCE), area under the retention curve (R-AUC), and error detection (E-Det). We state the metric definitions in Appendix F, where we also show that the NLL, BS, and Acc are coordinate-independent, and the ECE, MCE, R-AUC, and E-Det are coordinate-dependent. The results of this section are summarized on the left of Table 1. 3.2.1

Binary decision

We first consider a binary decision problem where an action in {0, 1} (e.g., {do not treat, treat}) is taken based on the model’s predicted probability f (x) = f ∈ [0, 1], x ∈ X for the positive class 3 This is essentially a stricter version of decomposable and non-decomposable losses [27], requiring the same instance function s for each component i. 4We write (f , r) for the prediction vector with value f at the ith position and values r at the remaining n − 1 positions. i i 5We generalize our analysis to multiclass classification in Appendix G.1.

4

(e.g., disease present). We assume zero cost for correct predictions and costs CFP , CFN > 0 for false positive and false negative predictions. Without loss of generality, we normalize the costs as cFP = CFP /(CFP + CFN ) (so cFN = 1 − cFP ). For ease of notation, we write c = cFP in the following. With this setup, the Bayes act is to select action 1 whenever f > c [28]. This allows us to write the pointwise utility of a prediction f with true label y ∈ Y for a fixed cost c ∈ (0, 1) as uc (f, y) = −y(1 − c)1f ≤c − (1 − y)c1f >c . The dataset-level utility Uc (f , y) is defined as the average pointwise utility. As a direct consequence, Uc is pointwise-separable. By Lemma 3.6, only the NLL, BS, and Acc can be decision-aligned for {Uc }c∈(0,1) . In fact, for the specific downstream utility uc , it is already known that PSRs l admit an integral representation (under mild conditions, see [14, 15, 16, 9, 10, 17]), Z 1 l(f, y) = −uc (f, y) π(c) dc. (3) 0

The expressions for π(c) for the NLL, BS, and Acc6 have been stated before (see, e.g., [16, 9]). Proposition 3.7 (Buja et al. [16]). The NLL, BS, and Acc are decision-aligned w.r.t. {Uc }c∈(0,1) with πNLL (c) = 1/(c(1 − c)), πBS = 2, πAcc (c) = 2δ0.5 (c) and hy = id for all y. While this decision-theoretic interpretation suggests that good performance on PSRs and Acc implies good decision-making, we would like to emphasize that the implicit priors of common PSRs are pathological.7 In many binary decisions, the cost of a false negative (e.g., missed disease) is much higher than the cost of a false positive (e.g., unnecessary treatment [28]). πNLL (c) places unbounded mass on extreme regions, rewarding performance where one error type is essentially free and a trivial policy suffices. The BS weighs each c ∈ (0, 1) equally, corresponding to an uninformative prior. πAcc (c) is degenerate, expressing the strong assumption that the costs are perfectly symmetric. We visualize πNLL , πBS , and πAcc on the left of Figure G.1. Since the ECE, MCE, R-AUC, and E-Det are coordinate-dependent, we can directly follow from Lemma 3.6 that they are not decision-aligned w.r.t. {Uc }c∈(0,1) . Corollary 3.8. The ECE, MCE, R-AUC, and E-Det are not decision-aligned w.r.t. {Uc }c∈(0,1) . 3.2.2

Top-k selection

Top-k selection for k ∈ [n − 1] := {1, . . . , n − 1} arises, for instance, in the context of recommender systems. As downstream utility, we use the established precision@k metric (in line with, e.g., [29]), 1 X Uk (f , y) = yi , k i∈Sk (f )

where Sk (f ) is the index set of the selected instances according to the predicted scores.8 This can be interpreted as receiving equal reward (normalized to k1 ) for each correctly selected positive instance, and no reward for selecting negative instances. The Bayes act is to select the k instances with the highest scores [30]. Uk is non-separable (whether an instance contributes to Uk depends on its rank relative to all other predictions, and therefore Uk cannot be written as a sum of independent terms), so we cannot leverage Lemma 3.6. Nevertheless, we are able to prove that none of the metrics we consider are decision-aligned w.r.t. {Uk }k∈[n−1] . Proposition 3.9. The NLL, BS, Acc, ECE, MCE, R-AUC, and E-Det are not decision-aligned w.r.t. {Uk }k∈[n−1] . Proof sketch. The value of Uk is only influenced by the subset Sk (f ), and all metrics we consider evaluate on the entire dataset. One can find predictions f 1 , f 2 that yield different metric values but leave the expected utility unchanged. We state the full proof in Appendix C.3. Remark 3.10 (Case k = n). When defining Uk , we excluded the trivial case of k = n (select all data points). Note, however, that Acc actually equals Uk for k = n. In other words, Acc is decision-aligned w.r.t. {Uk }k∈[n] under the (pathological) prior π(k) = δn (k). 6 Note that the pointwise Acc is not a PSR, but can still be expressed through (3), rather by coincidence than by principle. 7We characterize pathological priors as either (a) degenerate priors, (b) uninformative priors, or (c) priors that place unbounded mass on extreme regions where UQ is least needed. 8 For S (f ) to be uniquely defined, one can choose a tie-breaking rule or restrict to f with unique entries. k

5

3.3

Regression

In this section, we consider probabilistic univariate9 regression models. We assume a Gaussian predictive distribution10 and consider models of the form f : X → R × R>0 , mapping from the input space X to the predictive mean µ ∈ R and predictive variance σ 2 ∈ R>0 , so f = (µ, σ 2 ). The regression target is the true label y ∈ Y := R. As evaluation metrics, we consider the NLL, mean squared error (MSE), ECE, MCE, R-AUC, and E-Det. We state the metric definitions in Appendix F, where we also show that the NLL and MSE are coordinate-independent, and the ECE, MCE, R-AUC, and E-Det are coordinate-dependent. We summarize the results on the right of Table 1. 3.3.1

Selective prediction

In selective prediction, the decision-maker chooses an action in {predict, abstain}, where selecting predict corresponds to issuing a point prediction ŷ, while abstain corresponds to deferring the decision (e.g., to a human expert). We assume squared loss for point predictions and a fixed abstention cost λ ∈ R>0 . With this setup, the Bayes act is to issue the predictive mean µ whenever the predictive variance σ 2 is at most the abstention cost. This allows us to write the pointwise utility of a prediction f = (µ, σ 2 ) with true outcome y ∈ R for a fixed abstention cost λ as uλ (f, y) = −(µ − y)2 1σ2 ≤λ − λ 1σ2 >λ . Once again, the dataset-level utility Uλ (f , y) is defined as the average pointwise utility, so Uλ is pointwise-separable, and therefore the ECE, MCE, R-AUC, and E-Det are not decision-aligned w.r.t. {Uλ }λ∈R>0 by Lemma 3.6. We find that this also holds for the NLL and MSE. Proposition 3.11. The NLL, MSE, ECE, MCE, R-AUC, and E-Det are not decision-aligned w.r.t. {Uλ }λ∈R>0 . Proof sketch. The MSE only evaluates the point prediction µ and is therefore insensitive to σ 2 , so improvements in Uλ through σ 2 cannot be measured. The NLL does exhibit the same form as Uλ after monotone transformation under a Pareto prior π, but the expected utility diverges because π goes to infinity for λ → 0 too quickly, violating our requirements for a valid prior. For the coordinate-dependent metrics, we can apply Lemma 3.6. We state the full proof in Appendix C.4. Notably, the proof of Proposition 3.11 reveals that under minor additional assumptions, the NLL is decision-aligned w.r.t. the Uλ -family under a Pareto prior. Proposition 3.12. Fix any ε > 0 and define Pε := {f = (µ, σ 2 ) | σ 2 > ε}. Consider only predictions f ∈ Pεn . Then, the NLL is decision-aligned w.r.t. {Uλ }λ∈R>ε under π(λ) = ελ−2 and hy (m) = 2εm − ε log(2πε)

∀y.

Proof sketch. When restricting to λ, σi2 > ε for all i, we avoid the divergence issues from Proposition 3.11 and decision-alignment holds. We state the full proof in Appendix C.5. Note that this Pareto(1, ε) prior is pathological, as it places more weight on λ the smaller λ is. In other words, it anticipates almost negligible abstention costs. We visualize the Pareto prior on the right of Figure G.1, alongside the Dirac prior for the case λ = ∞ discussed below. Remark 3.13 (Case λ = ∞). When defining Uλ , we excluded the trivial case of λ = ∞ (always predict). Note, however, that the MSE actually equals Uλ for λ = ∞. In other words, the MSE is decision-aligned w.r.t. {Uλ }λ∈R>0 ∪{∞} under the (pathological) prior π(λ) = δ∞ (λ). 3.3.2

Top-k selection

Given predictions f = (µ, σ 2 ) and labels y, we consider a mean–variance trade-off of the form 1 X (yi − γσi2 ), Uϕ (f , y) = k i∈Sϕ (f )

9We generalize our analysis to multivariate regression in Appendix G.2. 10We assume a Gaussian distribution for simplicity, but any distribution with a well-defined mean and variance is admissible.

6

π(θ)

θ=c θ = k/n

10

mode

5 0

θ=λ θ = k/n θ=γ mode

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

utility parameter θ

0.4

0.6

0.8

1.0

utility parameter θ

Figure 1: Prior choices for our classification (left) and regression (right) prior-weighted utility metrics. where ϕ = (k, γ) ∈ Φ := [n − 1] × R>0 , and γ controls the degree of risk aversion. The Bayes act is to select the top-k instances according to the score µi − γσi2 . This yields a selection policy that trades off between predicted benefit and uncertainty. As in classification, we can show that none of the metrics we consider is decision-aligned w.r.t. {Uϕ }ϕ∈Φ . Proposition 3.14. The NLL, MSE, ECE, MCE, R-AUC, and E-Det are not decision-aligned w.r.t. {Uϕ }ϕ∈Φ . Proof sketch. As in classification, the value of the utility is only influenced by the top-k subset, and all metrics we consider evaluate on the entire dataset. We state the full proof in Appendix C.6.

4

Prior-weighted utility metrics

The key takeaway from Section 3 is that common UQ evaluation metrics either encode implausible decision beliefs or do not reflect decision utility at all. In other words, current UQ benchmarks silently optimize for the wrong downstream world. In pursuit of a better UQ evaluation protocol, we propose prior-weighted utility (PWU) metrics which are, by definition, decision-aligned and encode plausible prior beliefs about downstream decisions.11 4.1

PWU construction

To define a PWU metric, we select a decision family {Uθ }θ∈Θ and a plausible prior π such that EUθ∼π,y (f ) < ∞ for all (f , y). To align the PWU metric Mπ with {Uθ }θ∈Θ , we apply Equation (1) directly with hy = id for all y: Z Mπ (f , y) := −Uθ (f , y) π(θ) dθ. (4) Θ

We propose a PWU metric for each decision problem analyzed in Section 3, obtaining Z 1 Z ∞ Mπc (f , y) = −Uc (f , y) πc (c) dc, Mπλ (f , y) = −Uλ (f , y) πλ (λ) dλ, Mπk (f , y) =

0 n−1 X

0

−Uk (f , y) πk (k),

Mπϕ (f , y) =

Z ∞ n−1 X 0

k=1

−Uϕ (f , y) πϕ (ϕ) dϕ.

k=1

In Figure 1, we visualize the priors we choose for our experiments. In Appendix I, we justify them explicitly and provide some general guidance on prior elicitation. Note that constructing a PWU metric is a general method that can be applied to a broader set of decisions (and priors) than just those considered in this paper. Since each metric corresponds to a single decision family, it is advisable to use a variety of PWU metrics in UQ benchmarking to test models across different decision contexts. 4.2

Metric properties

The purpose of our PWU metrics is to evaluate UQ in terms of downstream decision utility. Thus, we intentionally defined them to be decision-aligned with their corresponding utility family. 11 Appendix H states a structured PWU documentation, similar to model cards [31], alongside a broader impact statement.

7

alignment (Kendall's τ)

Binary decision

Selective prediction

1

1

0

0

−1

−1

Top-k selection

Top-k selection

1

1

0

0

−1

−1

E E L S C et π c πk cc NL B A EC MC -AU E-D M M R not decision-aligned

E E E L C et π λ ϕ NL MS EC MC -AU E-D M M π R

pathological prior

plausible prior (ours)

Figure 2: Metric–utility alignment in classification (left) and regression (right), averaged over our five datasets. The coloring corresponds to our theoretical findings from Section 3. The PWU metrics align best with respect to their corresponding utility families. Proposition 4.1 (PWU metrics are decision-aligned). Pick any decision family {Uθ }θ∈Θ and a prior π over Θ. Then, the resulting PWU metric Mπ is decision-aligned w.r.t. {Uθ }θ∈Θ under π. Proof. Follows by comparing the PWU definition (4) to Definition 3.1 with hy = id for all y. Recall that decision-alignment is equivalent to strict order- and tie-preservation. Hence, PWU metrics offer a principled and decision-theoretic value for UQ benchmarking. Furthermore, using a PWU metric does not sacrifice properness—all PWU metrics are proper, and even strictly proper if the Bayes act for each Uθ is unique for all f . Proposition 4.2 (PWUR metrics are proper). Pick any decision family {Uθ }θ∈Θ and a corresponding prior π, and assume Θ EY ∼p [|Uθ,Y (f )|] π(θ) dθ < ∞ for all (f , y). Then, the resulting PWU metric Mπ is a PSR, and it is strictly proper if the Bayes acts for each Uθ are unique under all f . Proof. Mπ is decision-aligned by Proposition 4.1 and, by comparing (4) to Definition 3.1, we have hy = id for all y. Therefore, we can apply Proposition D.1 with a = 1, b = 0 and conclude that Mπ is proper (and strictly proper if the Bayes acts for each Uθ are unique under all f ). Our PWU metrics do not come without limitations (see also Appendix H). No single PWU can cover all downstream objectives, so we recommend using a variety of PWUs in UQ evaluation. Furthermore, PWU priors may be misspecified. However, in Appendix J.3 we show empirically that PWU metrics retain strong positive utility-alignment even under severe prior perturbation, in stark contrast to the brittleness of conventional metrics under their pathological implicit priors. These results highlight PWUs’ suitability for general-purpose UQ benchmarking.

5

Experiments

5.1

Controlled experiments on benchmark datasets

We implement ten binary classification and ten univariate regression models12 and evaluate these models on five datasets each. The experiments aim to measure how well the model ranking of each metric M aligns with the performance on downstream utilities. We use all metrics analyzed in Section 3 in addition to our PWU metrics, and compare their rankings to the four decision utilities 12We present results for the multiclass and multivariate settings in Appendix K.

8

Table 2: Metric–utility alignment (Kendall’s τ of model rankings) in the electricity market case study. We mark the highest median in bold. The bottom row states the 5th and 95th percentiles. The PWU metrics have the strongest and most stable positive bidding utility alignment. NLL

MSE

ECE

MCE

R-AUC

E-Det

Mπc

Mπ k

-0.16

0.09

-0.24

-0.24

-0.07

0.07

0.16

0.16

[-0.47, 0.25]

[-0.47, 0.64]

[-0.51, 0.12]

[-0.56, 0.16]

[-0.42, 0.38]

[-0.38, 0.56]

[-0.16, 0.38]

[-0.11, 0.42]

Uθ , θ ∈ {c, k, λ, ϕ} from Section 3. We then assess how well the model rankings of the metrics align with the model rankings of the utilities using Kendall’s τ . We repeat this procedure on 100 differently sampled test sets to quantify the results’ variability. In Figure 2, we report our results as boxplots13 and state the exact numbers in Tables J.1, J.2, J.3, J.4. We detail our chosen models, datasets, experimental setup, and general reproducibility in Appendix J. Overall, even though some conventional metrics are decision-aligned, we see that their pathological priors negatively impact their utility-alignment. Crucially, in top-k selection and selective prediction, most of the common metrics exhibit little to no utility alignment, rendering them largely uninformative for downstream decision-making. In contrast, our PWU metrics align best with their corresponding utility family. 5.2

Applied case studies

While the above experiments consider the same utilities as our PWUs, we now deviate from this setup and implement three applied case studies. In our first case study, we consider a wind farm operator who wants to sell electricity in the day-ahead market, where bids must be placed one day in advance. If the actual wind power production deviates from the bid, they risk a penalty for the resulting shortfall or surplus, as, in both cases, the electricity grid’s supply-demand balance needs to be restored. We follow the setup from Bruninx et al. [32], who derive an optimal bidding policy that resembles a selective prediction problem: bid only when the wind power forecast is sufficiently certain. The utility (payoff) is market-dependent for every bid/abstention and unknown in advance, deviating significantly from our controlled setup. We explain the case study in more detail in Appendix J.2 and show the results in Table 2 and Figure J.5. In this real-life setup, none of the conventional UQ metrics have a stable positive bidding utility alignment, again showcasing how disconnected generic metrics are from real-world tasks. In contrast, our PWU metrics exhibit the strongest utility alignment. In our remaining two case studies (see Appendix J.2), we consider the tasks of credit approval and peer-to-peer lending, again using real-world economic payoffs as utility functions. We observe once more that our PWU metrics reliably align with the decision utility—despite the fact that some decision parameters differ strongly from our chosen PWU priors (see Figure J.4). This demonstrates that PWU metrics remain robust under prior misspecification, a finding further supported by our systematic sensitivity analysis (see Appendix J.3).

6

Conclusion

We introduced decision-alignment as a criterion that connects uncertainty evaluation metrics to downstream decision utility and used it to diagnose that many standard UQ metrics are either misaligned with common decision problems or encode pathological implicit priors. Motivated by this insight, we proposed PWU metrics, a principled class of decision-aligned PSRs that enable utility-relevant benchmarking. Across benchmark datasets and real-world case studies, PWU metrics consistently produce model rankings that better reflect realized decision utility than conventional UQ metrics. Our results suggest that the dominant practice of evaluating UQ with generic metrics can be systematically misleading when the goal is to support downstream decisions. Instead, we claim that meaningful progress in probabilistic ML requires evaluating models with a variety of PWU metrics,14 so that UQ benchmarks bring methods to light that enable reliable decision-making. 13 The boxplots show the median, 25th and 75th percentiles via boxes, and 5th, 95th percentiles via whiskers. 14We provide our concrete recommendations for future UQ benchmarking that result from this paper in Appendix L.

9

Acknowledgments and Disclosure of Funding AS was supported by the DAAD program Konrad Zuse Schools of Excellence in Artificial Intelligence, sponsored by the Federal Ministry of Research, Technology and Space. VF was supported by the Branco Weiss Fellowship.

References [1] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua V. Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips. cc/paper_files/paper/2019/file/8558cb408c1d76621371888657d2eb1d-Paper.pdf. [2] Meet P Vadera, Jinyang Li, Adam D Cobb, Brian Jalaian, Tarek Abdelzaher, and Benjamin M Marlin. URSABench: A system for comprehensive benchmarking of bayesian deep neural network models and inference methods. In Proceedings of Machine Learning and Systems, volume 4, pages 217–237, 2022. URL https://proceedings.mlsys.org/paper_files/paper/2022/file/ 4a420924d20bc025ebb37849169e6ebd-Paper.pdf. [3] Luis Basora, Arthur Viens, Manuel Arias Chao, and Xavier Olive. A benchmark on uncertainty quantification for deep learning prognostics. Reliability Engineering & System Safety, 253:110513, 2025. URL https://linkinghub.elsevier.com/retrieve/pii/S0951832024005854. [4] Andrew J. Vickers and Elena B. Elkin. Decision curve analysis: A novel method for evaluating prediction models. Medical Decision Making, 26(6):565–574, 2006. URL https://journals.sagepub.com/ doi/10.1177/0272989X06295361. [5] Michael W. Dusenberry, Dustin Tran, Edward Choi, Jonas Kemp, Jeremy Nixon, Ghassen Jerfel, Katherine Heller, and Andrew M. Dai. Analyzing the role of model uncertainty for electronic health records. In Proceedings of the ACM Conference on Health, Inference, and Learning, pages 204–213. ACM, 2020. URL https://dl.acm.org/doi/10.1145/3368555.3384457. [6] Luciana Ferrer and Daniel Ramos. Evaluating posterior probabilities: Decision theory, proper scoring rules, and calibration. Transactions on Machine Learning Research, 2025. URL https://openreview. net/forum?id=qbrE0LR7fF. [7] Moritz Hardt and Benjamin Recht. Patterns, predictions, and actions: A story about machine learning, 2021. URL http://arxiv.org/abs/2102.05242. [8] Zachary Nado, Neil Band, Mark Collier, Josip Djolonga, Michael W. Dusenberry, Sebastian Farquhar, Qixuan Feng, Angelos Filos, Marton Havasi, Rodolphe Jenatton, Ghassen Jerfel, Jeremiah Liu, Zelda Mariet, Jeremy Nixon, Shreyas Padhy, Jie Ren, Tim G. J. Rudner, Faris Sbahi, Yeming Wen, Florian Wenzel, Kevin Murphy, D. Sculley, Balaji Lakshminarayanan, Jasper Snoek, Yarin Gal, and Dustin Tran. Uncertainty baselines: Benchmarks for uncertainty & robustness in deep learning, 2022. URL http://arxiv.org/abs/2106.04015. [9] Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359–378, 2007. URL http://www.tandfonline. com/doi/abs/10.1198/016214506000001437. [10] Niko Brümmer. Measuring, refining and calibrating speaker and language information extracted from speech, 2010. URL https://scispace.com/pdf/ measuring-refining-and-calibrating-speaker-and-language-53glvzuiuj.pdf. PhD Thesis. [11] Romain Pic, Clément Dombry, Philippe Naveau, and Maxime Taillardat. Proper scoring rules for multivariate probabilistic forecasts based on aggregation and transformation. Advances in Statistical Climatology, Meteorology and Oceanography, 11(1):23–58, 2025. URL https://ascmo.copernicus.org/ articles/11/23/2025/. [12] Ian Osband, Zheng Wen, Seyed Mohammad Asghari, Brendan O’Donoghue, Botao Hao, Dieterich Lawson, Morteza Ibrahimi, Xiuyuan Lu, Vikranth Dwaracherla, and Benjamin Van Roy. The neural testbed: Evaluating joint predictions. In Proceedings of the 36th International Conference on Neural Information Processing Systems, volume 35 of NIPS ’22, pages 12554–12565. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ 5141f6bc105d30edbae48f1d2e0b1e66-Paper-Conference.pdf.

10

[13] Alexander Philip Dawid and Monica Musio. Theory and applications of proper scoring rules. METRON, 72(2):169–183, 2014. URL http://link.springer.com/10.1007/s40300-014-0039-y. [14] Leonard J. Savage. Elicitation of personal probabilities and expectations. Journal of the American Statistical Association, 66(336):783–801, 1971. ISSN 0162-1459, 1537-274X. doi: 10.1080/01621459.1971. 10482346. URL http://www.tandfonline.com/doi/abs/10.1080/01621459.1971.10482346. [15] Mark J. Schervish. A general method for comparing probability assessors. The Annals of Statistics, 17(4): 1856–1879, 1989. URL https://www.jstor.org/stable/2241668. [16] Andreas Buja, Werner Stuetzle, and Yi Shen. Loss functions for binary class probability estimation and classification: Structure and applications, 2005. URL http://stat.wharton.upenn.edu/~buja/ PAPERS/paper-proper-scoring.pdf. [17] Mark D. Reid and Robert C. Williamson. Information, divergence and risk for binary experiments. Journal of Machine Learning Research, 12:731–817, 2011. URL https://jmlr.csail.mit.edu/papers/ v12/reid11a.html. [18] Werner Ehm, Tilmann Gneiting, Alexander Jordan, and Fabian Krüger. Of quantiles and expectiles: Consistent scoring functions, choquet representations and forecast rankings. Journal of the Royal Statistical Society Series B: Statistical Methodology, 78(3):505–562, 2016. URL https://academic.oup.com/ jrsssb/article/78/3/505/7040984. [19] Max-Heinrich Laves, Sontje Ihler, Jacob F Fast, Luder A Kahrs, and Tobias Ortmaier. Well-calibrated regression uncertainty in medical imaging with deep learning. In Proceedings of the Third Conference on Medical Imaging with Deep Learning, volume 121, pages 393–412. PMLR, 2020. URL https: //proceedings.mlr.press/v121/laves20a.html. [20] Ruotao Zhang, Constantine Gatsonis, and Jon Arni Steingrimsson. Role of calibration in uncertaintybased referral for deep learning. Statistical Methods in Medical Research, 32(5):927–943, 2023. URL https://journals.sagepub.com/doi/10.1177/09622802231158811. [21] Sebastian G. Gruber and Florian Buettner. Better uncertainty calibration via proper scores for classification and beyond. In 36th Conference on Neural Information Processing Systems, 2022. URL http://arxiv. org/abs/2203.07835. [22] Robert Kleinberg, Renato Paes Leme, Jon Schneider, and Yifeng Teng. U-calibration: Forecasting for an unknown agent. In Proceedings of Machine Learning Research, volume 195, pages 1–3, 2023. URL https://proceedings.mlr.press/v195/kleinberg23a/kleinberg23a.pdf. [23] Lunjia Hu and Yifan Wu. Calibration error for decision making, 2024. URL http://arxiv.org/abs/ 2404.13503. [24] Mingda Qiao and Eric Zhao. Truthfulness of decision-theoretic calibration measures, 2025. URL http://arxiv.org/abs/2503.02384. [25] Mochen Yang and Xuan Bi. Cost-aware calibration of classifiers. INFORMS Journal on Data Science, 4 (2):101–113, 2025. URL https://pubsonline.informs.org/doi/10.1287/ijds.2024.0038. [26] Mahmoud Hegazy, Michael I. Jordan, and Aymeric Dieuleveut. Scalable utility-aware multiclass calibration, 2025. URL http://arxiv.org/abs/2510.25458. [27] Shu Hu, Xin Wang, and Siwei Lyu. Rank-based decomposable losses in machine learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–20, 2023. URL https: //ieeexplore.ieee.org/document/10184478/. [28] Charles Elkan. The foundations of cost-sensitive learning. In Proceedings of the 17th International Joint Conference on Artificial Intelligence, volume 2 of IJCAI’01, pages 973–978. Morgan Kaufmann Publishers Inc., 2001. URL https://dl.acm.org/doi/abs/10.5555/1642194.1642224. [29] Yan-Martin Tamm, Rinchin Damdinov, and Alexey Vasilev. Quality metrics in recommender systems: Do we calculate metrics consistently? In Fifteenth ACM Conference on Recommender Systems, pages 708–713, 2021. URL http://arxiv.org/abs/2206.12858. [30] Marek Wydmuch, Kalina Jasinska, Mikhail Kuznetsov, Róbert Busa-Fekete, and Krzysztof Dembczyński. A no-regret generalization of hierarchical softmax to extreme multi-label classification. In 32nd Conference on Neural Information Processing Systems, 2018. URL http://arxiv.org/abs/1810.11671.

11

[31] Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* ’19, page 220–229, New York, NY, USA, 2019. Association for Computing Machinery. URL https://doi.org/10.1145/3287560.3287596. [32] Max Bruninx, Timothy Verstraeten, Jalal Kazempour, and Jan Helsen. Day-ahead bidding strategies for wind farm operators under a one-price balancing scheme. In Proceedings of the 16th ACM International Conference on Future and Sustainable Energy Systems, pages 719–726. ACM, 2025. URL https: //dl.acm.org/doi/10.1145/3679240.3734639. [33] Viktor Bengs, Eyke Hüllermeier, and Willem Waegeman. Pitfalls of epistemic uncertainty quantification through loss minimisation. In Advances in Neural Information Processing Systems, volume 35, pages 29205–29216. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_ files/paper/2022/file/bc1d640f841f752c689aae20b31198c1-Paper-Conference.pdf. [34] Viktor Bengs, Eyke Hüllermeier, and Willem Waegeman. On second-order scoring rules for epistemic uncertainty quantification. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pages 2078–2091. PMLR, 23–29 Jul 2023. URL https://proceedings.mlr.press/ v202/bengs23a.html. [35] Shireen Kudukkil Manchingal, Muhammad Mubashar, Kaizheng Wang, Keivan K1 Shariatmadar, and Fabio Cuzzolin. Random-set neural networks. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=pdjkikvCch. [36] Shireen Kudukkil Manchingal, Andrew Bradley, Julian F. P. Kooij, Keivan K1 Shariatmadar, Neil YorkeSmith, and Fabio Cuzzolin. Epistemic artificial intelligence is essential for machine learning models to truly’know when they do not know’, 2025. URL https://arxiv.org/abs/2505.04950. [37] A. Allen Bradley and Stuart S. Schwartz. Summary verification measures and their interpretation for ensemble forecasts. Monthly Weather Review, 139(9):3075–3089, 2011. URL https://journals. ametsoc.org/doi/10.1175/2010MWR3305.1. [38] Novin Shahroudi, Viacheslav Komisarenko, and Meelis Kull. Aligning the evaluation of probabilistic predictions with downstream value, 2025. URL http://arxiv.org/abs/2508.18251. [39] Rabanus Derr and Robert C. Williamson. Forecast evaluation and the relationship of regret and calibration, 2025. URL http://arxiv.org/abs/2401.14483. [40] Gerardo A Flores, Alyssa H Smith, Julia A Fukuyama, and Ashia C Wilson. Aligning evaluation with clinical priorities: Calibration, label shift, and error costs. In The 39th Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=b7Ka31HvXU. [41] Gerardo Flores, Abigail Schiff, Alyssa H. Smith, Julia A. Fukuyama, and Ashia C. Wilson. A consequentialist critique of binary classification evaluation: Theory, practice, and tools, 2026. URL http://arxiv.org/abs/2504.04528. [42] Peng Cui, Wenbo Hu, and Jun Zhu. Calibrated reliable regression using maximum mean discrepancy. In Proceedings of the 34th International Conference on Neural Information Processing Systems, volume 33 of NIPS ’20, pages 17164–17175. Curran Associates, Inc., 2020. URL https://proceedings.neurips. cc/paper_files/paper/2020/file/c74c4bf0dad9cbae3d80faa054b7d8ca-Paper.pdf. [43] Andrey Malinin, Neil Band, Ganshin, Alexander, German Chesnokov, Yarin Gal, Mark J. F. Gales, Alexey Noskov, Andrey Ploskonosov, Liudmila Prokhorenkova, Ivan Provilkov, Vatsal Raina, Vyas Raina, Roginskiy, Denis, Mariya Shmatova, Panos Tigas, and Boris Yangel. Shifts: A dataset of real distributional shift across multiple large-scale tasks. In 35th Conference on Neural Information Processing Systems Track on Datasets and Benchmmarks, 2021. URL https://openreview.net/forum?id=qM45LHaWM6E. [44] Victor Bouvier, Simona Maggio, Alexandre Abraham, and Léo Dreyfus-Schmidt. Towards clear expectations for uncertainty estimation, 2022. URL http://arxiv.org/abs/2207.13341. [45] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pages 1321–1330. PMLR, 2017. URL https://proceedings.mlr.press/v70/guo17a.html. [46] Petrus Mikkola, Osvaldo A. Martin, Suyog Chandramouli, Marcelo Hartmann, Oriol Abril Pla, Owen Thomas, Henri Pesonen, Jukka Corander, Aki Vehtari, Samuel Kaski, Paul-Christian Bürkner, and Arto Klami. Prior knowledge elicitation: The past, present, and future. Bayesian Analysis, 19(4):1129–1161, 2024. URL https://doi.org/10.1214/23-BA1381.

12

[47] James Hensman, Alexander Matthews, and Zoubin Ghahramani. Scalable variational gaussian process classification. In Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics, volume 38, pages 351–360. PMLR, 2015. URL https://proceedings.mlr.press/v38/ hensman15.html. [48] Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. TabPFN: A transformer that solves small tabular classification problems in a second. In The 11th International Conference on Learning Representations, 2023. URL https://openreview.net/pdf?id=cp5PvcI6w8_. [49] Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. In Advances in Neural Information Processing Systems, volume 34, pages 18932– 18943. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/ paper/2021/file/9d86d83f925f2149e9edb0ac3b49229c-Paper.pdf. [50] Gowthami Somepalli, Avi Schwarzschild, Micah Goldblum, C. Bayan Bruss, and Tom Goldstein. SAINT: Improved neural networks for tabular data via row attention and contrastive pre-training. In NeurIPS 2022 First Table Representation Workshop, 2022. URL https://openreview.net/forum? id=FiyUTAy4sB8. [51] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/ paper/2017/file/9ef2ed4b7fd2c810847ffa5fa85bce38-Paper.pdf. [52] Tony Duan, Anand Avati, Daisy Yi Ding, Khanh K Thai, Sanjay Basu, Andrew Ng, and Alejandro Schuler. NGBoost: Natural gradient boosting for probabilistic prediction. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research. PMLR, 2020. URL https://proceedings.mlr.press/v119/duan20a.html. [53] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830, 2011. URL https://jmlr.org/papers/v12/pedregosa11a.html. [54] Jacob R Gardner, Geoff Pleiss, David Bindel, Kilian Q Weinberger, and Andrew Gordon Wilson. Gpytorch: Blackbox matrix-matrix gaussian process inference with gpu acceleration. In Advances in Neural Information Processing Systems, 2018. URL https://proceedings.neurips.cc/paper/2018/hash/ 27e8e17134dd7083b050476733207ea1-Abstract.html. [55] Markelle Kelly, Rachel Longjohn, and Kolby Nottingham. The UCI machine learning repository, 2025. URL https://archive.ics.uci.edu. [56] S Moro, P Rita, and P Cortez. Bank marketing, 2014. URL https://doi.org/10.24432/C5K306. [57] Andras Janosi, William Steinbrunn, Matthias Pfisterer, and Robert Detrano. Heart disease, 1989. URL https://doi.org/10.24432/C52P4X. [58] V Sigillito, S Wing, L Hutton, and K Baker. Ionosphere, 1989. URL https://doi.org/10.24432/ C5W01B. [59] Jeff Schlimmer. Mushroom, 1981. URL https://doi.org/10.24432/C5959T. [60] T Sejnowski and R Gorman. Connectionist bench (sonar, mines vs. rocks), 1988. URL https://doi. org/10.24432/C5T01Q. [61] S Vito. Air quality, 2008. URL https://doi.org/10.24432/C59K5F. [62] R Quinlan. Auto MPG, 1993. URL https://doi.org/10.24432/C5859H. [63] Athanasios Tsanas and Angeliki Xifara. Energy efficiency, 2012. URL https://doi.org/10.24432/ C51307. [64] P Tfekci and H Kaya. Combined cycle power plant, 2014. URL https://doi.org/10.24432/C5002N. [65] P Cortez, A Cerdeira, F Almeida, T Matos, and J Reis. Wine quality, 2009. URL https://doi.org/10. 24432/C56S3T. [66] Elia. Elia open data platform, 2025. URL https://opendata.elia.be.

13

[67] European Network of Transmission System Operators for Electricity. ENTSO-e transparency platform, 2025. URL https://transparency.entsoe.eu. [68] Alejandro Correa Bahnsen, Djamia Aouada, and Bjorn Ottersten. Example-dependent cost-sensitive logistic regression for credit scoring. In 13th International Conference on Machine Learning and Applications, pages 263–269. IEEE, 2014. URL http://ieeexplore.ieee.org/document/7033125/. [69] PAKDD. Credit risk assessment on a private label credit card application, 2009. URL https://pakdd. org/archive/pakdd2009/front/show/competition.html. Data Mining Competition. [70] Will Cukierski. GiveMeSomeCredit, 2011. GiveMeSomeCredit. Credit Fusion. [71] Alejandro Correa Bahnsen. costcla, CostSensitiveClassification.

2016.

URL https://kaggle.com/competitions/ URL https://github.com/albahnsen/

[72] Ajay Byanjankar, József Mezei, and Markku Heikkilä. Data-driven optimization of peer-to-peer lending portfolios based on the expected value framework. Intelligent Systems in Accounting, Finance and Management, 28(2):119–129, 2021. URL https://onlinelibrary.wiley.com/doi/10.1002/isaf. 1490. [73] George Nathan. Lending club loan data, 2020. wordsforthewise/lending-club.

URL https://www.kaggle.com/datasets/

[74] J Blackard. Covertype, 1998. URL https://doi.org/10.24432/C50K5N. [75] M Koklu and Ilker Ali Özkan. Dry bean, 2020. URL https://doi.org/10.24432/C50S4B. [76] R Fisher. Iris, 1936. URL https://doi.org/10.24432/C56C76. [77] E Alpaydin and Fevzi Alimoglu. Pen-based recognition of handwritten digits, 1996. URL https: //doi.org/10.24432/C5MG6K. [78] Stefan Aeberhard and M Forina. Wine, 1992. URL https://doi.org/10.24432/C5PC7J. [79] Gary Bradshaw. Solar flare, 1989. URL https://doi.org/10.24432/C5530G. [80] Enrique Paredes and Rafael Ballester-Ripoll. SGEMM GPU kernel performance, 2017. URL https: //doi.org/10.24432/C5MK70. [81] Athanasios Tsanas and Max Little. Parkinsons telemonitoring, 2009. URL https://doi.org/10. 24432/C5ZS3N.

14

A

Scope of the paper

This appendix clarifies the scope of our work, which we term first-order UQ evaluation. Throughout this paper, a probabilistic prediction refers to a single distribution over the label space; we call such an object a first-order probabilistic prediction. In probabilistic binary classification, this is a single f ∈ [0, 1] encoding Pr(Y = 1 | X); in regression, it is a single distribution over Y such as a Gaussian N (µ, σ 2 ). This is the object almost universally produced and evaluated in probabilistic ML benchmarks [1, 8, 3], and the object for which all metrics we analyze (NLL, BS, MSE, Acc, ECE, MCE, R-AUC, E-Det) are defined. A separate strand of the literature studies second-order probabilistic predictions: objects that take the form of a distribution over distributions. There are several reasons we restrict our scope to first-order evaluation. First, our work is positioned as a response to the current mainstream UQ evaluation literature (see Sections 1 and 2), which consistently uses metrics defined on first-order probabilistic predictions. Second, there is no broadly accepted evaluation protocol for second-order UQ to begin with [33, 34]. Many studies even evaluate the marginal predictive (see, e.g., [35, 36]), which is again first-order UQ and our analysis readily applies. Third, even if future research shifts more strongly toward second-order probabilistic predictions—a development we would welcome— we expect first-order evaluation to remain important for two reasons: interpretability—a single predictive distribution is substantially easier to communicate, visualize, and reason about than a distribution over distributions; and decision-theoretic use—Bayes-optimal actions under expected utility maximization are defined with respect to a single predictive distribution. Even models that maintain second-order beliefs internally are typically marginalized to a first-order posterior predictive distribution at decision time, so first-order quality remains the immediate driver of realized decision utility in most downstream pipelines. Nevertheless, extending decision-aligned evaluation to second-order predictions is an interesting and important direction which we leave for future work.

B

Omitted related work

Ehm et al. [18] study the evaluation and ranking of point forecasts through consistent scoring functions for functionals (e.g., quantiles). Similar to our decision-alignment, they consider consistency towards the functional of interest. They prove an order sensitivity result that resembles our Proposition 3.4, however, they order point forecasts and associated scores, while we order scores and utilities of distributional reports. In addition, they show only one direction (consistency implies order sensitivity) and consider only inequalities, whereas we show that decision-alignment is equivalent to strict order and tie preservation. Further, Ehm et al. [18] also discuss the decision-theoretic interpretation of binary decisions for PSRs and propose to use Murphy diagrams to evaluate and rank forecasts when the downstream task is unknown. As in our approach, they do not commit to a single decision parameter θ. Murphy diagrams plot the score value over varying θ, whereas we set plausible prior π(θ) and report the corresponding (scalar) integral value. In addition, we do not restrict ourselves to an inherent downstream task like binary decisions in binary classification, but offer several decision-specific metrics for several tasks. Bradley and Schwartz [37] also introduce a general framework tailored to ensemble weather forecasts, which is used to interpret known metrics. In their setup, probabilistic forecasts f for a continuous variable Y are mapped to probability forecasts for threshold events of the form Y ≤ y, and forecast quality is studied as a function Q(y) of the threshold. Within this framework, several standard verification scores (e.g., the continuous ranked probability skill score) are shown to correspond to weighted averages of the forecast quality function over thresholds. Our decision-alignment framework is more general, allowing metric interpretation with respect to any given (integrable) utility family and is theoretically motivated through strict order and tie preservation in terms of expected downstream utility. Shahroudi et al. [38] propose to learn a proxy evaluation function for a known downstream task, motivated by the fact that evaluating the downstream utility can be costly. While their framework is termed evaluation alignment, similarly to our decision-alignment, the concepts differ substantially: In evaluation alignment, the decision loss needs to be known and is approximated with a learned 15

PSR, while we propose general-purpose metrics for general-purpose evaluation where no specific downstream task is anticipated. Ferrer and Ramos [6] advocate the use of expected PSRs (ESPRs), which are also defined as weighted integrals over a cost function (i.e., negative utility), but differ from our work in the sense that Ferrer and Ramos [6] only consider classification with pointwise classification cost. They neither analyze other metrics in terms of such an integral representation nor propose concrete new weight functions (i.e., priors), nor consider other utilities. The focus of the paper is rather to demonstrate that such ESPRs are superior to calibration metrics for UQ evaluation. Pic et al. [11] propose another metric framework for probabilistic weather forecasts, specific to PSRs, based on aggregation (combining PSRs through a finite weighted sum) and transformation (transform forecasts and labels before applying the PSR). This is a different framework than decisionalignment—we do not require a transformation and our aggregation step (prior-weighted integral) is more general. Also, Derr and Williamson [39] introduce a general framework for expressing evaluation metrics that represents metrics as the average realized value of gambles drawn from a predefined set. Their framework aims to unify a wide range of existing evaluation metrics by elucidating their structural relationships. In contrast, our work is not primarily concerned with unification, but with analyzing when and how metrics are informative for downstream decision-making. Flores et al. [40] also build on the Schervish representation [15] to propose evaluation metrics for binary classifiers that reflect clinical priorities. Similarly, Flores et al. [41] propose to use a clipped variant of the Brier score to evaluate binary classifiers to better reflect decision-relevant thresholds. These works are conceptually a special case of our PWU metrics.

C

Omitted proofs

C.1

Proof of Proposition 3.4

Proposition 3.4 (Decision-alignment ⇔ strict order- and tie-preservation). Fix a decision family {Uθ }θ∈Θ and a prior π. Then, for any fixed y, the following statements are equivalent: (I) M is decision-aligned w.r.t. {Uθ }θ∈Θ under π. (II) M is strictly order-preserving and tie-preserving for {Uθ }θ∈Θ under π, i.e., for all f 1 , f 2 , My (f 1 ) < My (f 2 ) ⇒ EUθ∼π,y (f 1 ) < EUθ∼π,y (f 2 ), My (f 1 ) = My (f 2 ) ⇒ EUθ∼π,y (f 1 ) = EUθ∼π,y (f 2 ). Proof. Fix any y. (I)⇒(II): Since (I) holds, there exists a strictly increasing hy s.t. hy (My (f )) = EUθ∼π,y (f ) for all f . Take any f 1 , f 2 . If My (f 1 ) < My (f 2 ), then strict monotonicity of hy implies hy (My (f 1 )) < hy (My (f 2 )), hence EUθ∼π,y (f 1 ) < EUθ∼π,y (f 2 ). If My (f 1 ) = My (f 2 ), then, clearly, hy (My (f 1 )) = hy (My (f 2 )), hence EUθ∼π,y (f 1 ) = EUθ∼π,y (f 2 ). Altogether, (II) holds. (II)⇒(I): Assume (II) holds. Define hy on Im(My ) by hy (m) := EUθ∼π,y (f )

for any f with My (f ) = m. ′

This is well-defined: if My (f ) = My (f ), then tie preservation yields EUθ∼π,y (f ) = EUθ∼π,y (f ′ ). Moreover, for any f , by construction hy (My (f )) = EUθ∼π,y (f ). It remains to show that hy is strictly increasing. Let m1 < m2 be in Im(My ), and choose f 1 , f 2 s.t. My (f 1 ) = m1 and My (f 2 ) = m2 . Then, strict order preservation implies EUθ∼π,y (f 1 ) < EUθ∼π,y (f 2 ), i.e., hy (m1 ) < hy (m2 ). Hence, hy is strictly increasing, completing the proof. C.2

Proof of Lemma 3.6

Lemma 3.6 (Separability barrier). If the utility Uθ is pointwise-separable for every θ ∈ Θ and θ 7→ Uθ (f , y) is integrable for all (f , y), then any metric M that is decision-aligned w.r.t. {Uθ }θ∈Θ (for some prior π) must be coordinate-independent. 16

Proof. Fix any y and let M be decision-aligned w.r.t. {Uθ }θ∈Θ for some prior weight π. Denote the instance function of Uθ by uθ . By the assumed integrability of θ 7→ Uθ (f , y) for all (f , y) and the fact that the sum in (2) is finite, we may interchange summation in (2) and integration in (1) (Fubini/Tonelli): Z n n Z
1X 1X hy My (f ) = − uθ (fi , yi )π(θ)dθ = −uθ (fi , yi )π(θ)dθ. n i=1 n i=1 Θ Θ R Define the instance function s(fi , yi ) := Θ −uθ (fi , yi )π(θ)dθ, then the above equation reads n
1X s(fi , yi ). hy My (f ) = n i=1

Pn Define Sy (f ) := n1 i=1 s(fi , yi ), which is (by the form of its definition) pointwise-separable and hence also coordinate-independent, so, for a, b ∈ P and r, r ′ ∈ P n−1 , Sy ((a, r)) < Sy ((b, r)) ⇔ Sy ((a, r ′ )) < Sy ((b, r ′ )),
and because hy My (f ) = Sy (f ),



hy My ((a, r)) < hy My ((b, r)) ⇔ hy My ((a, r ′ )) < hy My ((b, r ′ )) . Since hy is strictly increasing, it preserves orderings, and the same coordinate-independence property must hold for My . The same statements follow analogously for equality and strict inequality in the opposite direction. C.3

Proof of Proposition 3.9

Proposition 3.9. The NLL, BS, Acc, ECE, MCE, R-AUC, and E-Det are not decision-aligned w.r.t. {Uk }k∈[n−1] . Proof. For any metric M ∈ {NLL, BS, Acc, ECE, MCE, R-AUC, E-Det}, assume for contradiction that there exists a prior π : [n − 1] → R≥0 s.t. for any fixed labels y, there exists a strictly increasing function hy : Im(My ) → R s.t. X
n−1 hy My (f ) = −Uk,y (f )π(k)

∀f .

(5)

k=1

The value of Uk is only influenced by the subset Sk (f ), and we seek predictions f 1 , f 2 that change our (monotonically transformed) metrics but not Uk . For this, consider the label vector y = 0 = (0, . . . , 0), because then, for every k ∈ [n − 1] and every f , Uk,0 (f ) = 0, and (5) would imply that
h0 M0 (f ) = 0 for all f . (6) Because h0 is strictly increasing, it is injective, and therefore (6) implies that M0 (f ) is constant in f : for all f 1 , f 2 , M0 (f 1 ) = M0 (f 2 ). (7) We now exhibit f 1 , f 2 s.t. (7) fails. Case M ∈ {NLL, BS, Acc}: Let f 1 = ( 41 , . . . , 14 ) and f 2 = ( 43 , . . . , 34 ). Remember that we consider the negative conventional accuracy as Acc. Then, NLL0 (f 1 ) = − log( 43 )

̸=

− log( 14 ) = NLL0 (f 2 ),

1 BS0 (f 1 ) = 16

̸=

9 16 = BS0 (f 2 ),

Acc0 (f 1 ) = −1

̸=

0 = Acc0 (f 2 ),

so (7) is violated. Case M ∈ {ECE, MCE}: Fix any binning scheme B = {B1 , . . . , Bm } that partitions [0, 1] and let f 1 , f 2 be as above. Then all entries of f j fall into a single bin Bj containing the constant value. Since y = 0, we have freq(Bj ) = 0 and conf(Bj ) equals the constant value. Therefore, ECE0 (f 1 ) = |0 − 12 | = 14 17

̸=

3 4 = ECE0 (f 2 ),

contradicting (7). Moreover, since only one bin is nonempty, the same holds for the MCE. Case M = R-AUC: Let f 1 , f 2 be as above. We have Acc0 (f 1,i:i∈R ) = −1 for every retained set R, hence R-AUC0 (f 1 ) = 0. For f 2 , we have Acc0 (f 2,i:i∈R ) = 0 for every R, hence R-AUC0 (f 2 ) = 1, contradicting (7). Case M = E-Det: Note that for the E-Det with y = 0, we have ei = 1 iff fi > 0.5. Choose numbers a, b ∈ (0, 0.5) with a < b, and define a′ := 1 − a, b′ := 1 − b so that a′ , b′ ∈ (0.5, 1) and s(a) = s(a′ ), s(b) = s(b′ ) by symmetry of entropy. Define two prediction vectors: f 1 = ( a, b′ , b′ , . . . , b′ ),

f 2 = ( b, a′ , a′ , . . . , a′ ).

Then, for f 1 , we have exactly one index with fi ≤ 0.5 (hence ei = 0) and n − 1 indices with fi > 0.5 (hence ei = 1). Moreover, the unique ei = 0 point has uncertainty score s(a), while every ei = 1 point has uncertainty score s(b′ ) = s(b). Since a < b and s(·) is strictly increasing on (0, 0.5) for entropy, we have s(a) < s(b), so all positives (e = 1) have strictly larger scores than all negatives (e = 0). Therefore, the AUROC of using si to predict ei satisfies E-Det0 (f 1 ) = −1. For f 2 , we again have exactly one index with ei = 0 and n − 1 indices with ei = 1. Now the unique ei = 0 point has uncertainty score s(b), while every ei = 1 point has uncertainty score s(a′ ) = s(a). Thus s(b) > s(a) implies that the single negative has strictly larger score than all positives, so E-Det0 (f 2 ) = 0, contradicting (7). In all cases, we reach a contradiction. Therefore, no such prior π can exist, which proves the claim. C.4

Proof of Proposition 3.11

Proposition 3.11. The NLL, MSE, ECE, MCE, R-AUC, and E-Det are not decision-aligned w.r.t. {Uλ }λ∈R>0 . Proof. For M ∈ {ECE, MCE, R-AUC, E-Det}, the claim follow directly by Lemma 3.6. For M ∈ {NLL, MSE}, assume for contradiction that there exists a prior π : R>0 → R≥0 s.t. for any fixed labels y there exists a strictly increasing function hy : Im(My ) → R s.t. Z
hy My (f ) = −Uλ,y (f )π(λ)dλ ∀f . (8) R>0

Case M = MSE: Fix any labels y and consider two prediction vectors f 1 = (µ, σ 2 1 ), f 2 = (µ, σ 2 2 ) with identical predictive means µ = y. Take any interval (a, b) ⊂ R>0 with Rb 2 2 π(λ)dλ > 0 (must exist for a prior π) and consider σ1,i = a < b = σ2,i for all i. Then a MSE(f 1 , y)
= MSE(f 2 , y)
= 0, since the MSE ignores predictive uncertainty entirely, so hy My (f 1 ) = hy My (f 2 ) . For the difference on the right-hand side of (8) when changing from f 1 to f 2 , we have Z Z −Uλ,y (f 2 )π(λ)dλ − −Uλ,y (f 1 )π(λ)dλ R>0

R>0

Z b

Z a λπ(λ)dλ + 0 −

= 0

Z b λπ(λ)dλ − 0 = λ

0

π(λ)dλ ̸= 0, a

contradicting (8). Case M = NLL: Fix any labels y. Step 1: Expected utility under π. For each coordinate i, −Uλ,yi (fi ) = (µi − yi )2 1σi2 ≤λ + λ1σi2 >λ . By linearity of integration, ! Z Z ∞ Z σi2 n 1X 2 −Uλ,y (f )π(λ)dλ = (µi − yi ) π(λ)dλ + λ π(dλ) . (9) n i=1 σi2 0 Define S(s) :=

R∞ s

π(λ)dλ, s > 0. 18

Step 2: One-dimensional reduction. Fix an index i. Fix all coordinates (µj , σj2 ) for j ̸= i and fix σi2 = s ∈ (0, ∞). Let x := (µi − yi )2 ∈ [0, ∞). Under these choices, My (f ) = As x + Bs ,

As :=

where Bs is independent of x, and from (9), Z −Uλ,y (f )π(λ)dλ = Cs x + Ds ,

1 > 0, 2ns

Cs :=

(10)

1 S(s) ≥ 0, n

(11)

with Ds independent of x. Substituting (10)–(11) into (8) yields h(As x + Bs ) = Cs x + Ds

∀x ≥ 0.

(12)

Step 3: Form of h. Note the following: For A > 0, B ∈ R, and h a function defined on [B, ∞), we have that if there exist constants C, D s.t. h(Ax + B) = Cx + D ∀x ≥ 0, then   C C h(z) = z + D − B ∀z ∈ [B, ∞). A A This is because for any z ≥ B, substituting x = (z − B)/A ≥ 0 gives   C C h(z) = Cx + D = z + D − B . A A Applying this to (12), we conclude that h(z) = ms z + bs

∀z ≥ Bs ,

where

Cs = 2s S(s) ≥ 0. (13) As Because h is strictly increasing on im(My ) and [Bs , ∞) ⊆ im(My ), we must have ms > 0, hence S(s) > 0 for all s > 0. ms :=

Step 4: Consistency across different variances. Let s1 , s2 > 0. Repeating the above construction yields representations h(z) = ms1 z + bs1

(z ≥ Bs1 ),

h(z) = ms2 z + bs2

(z ≥ Bs2 ).

The half-lines [Bs1 , ∞) and [Bs2 , ∞) overlap, and both expressions describe the same function h on the overlap. Therefore, ms1 = ms2 . Since s1 , s2 are arbitrary, there exists a constant m > 0 s.t. ms = m ∀s > 0. Combining with (13) gives m S(s) = ∀s > 0. (14) 2s Step 5: Contradiction. From (14), we obtained Z ∞ m S(s) = π(λ)dλ = 2s s

∀s > 0,

for some constant m > 0. Since S is absolutely continuous with derivative S ′ (s) = −π(s) for a.e. s > 0, differentiating yields m π(s) = −S ′ (s) = 2 for a.e. s > 0. 2s But then, for any fixed s > 0, Z s Z s Z m m s1 λπ(λ)dλ = λ · 2 dλ = dλ = +∞. 2λ 2 0 λ 0 0 Consequently, the expected (negative) utility in (9) is infinite for any prediction vector with σi2 = s for some coordinate i, and thus the right-hand side of (8) is not well-defined, contradicting the definition of decision-alignment (which requires the integral to be well-defined for all admissible f ). Therefore, no such nonnegative measurable π can exist, and the regression NLL is not decision-aligned w.r.t. {Uλ }λ>0 . 19

C.5

Proof of Proposition 3.12

Proposition 3.12. Fix any ε > 0 and define Pε := {f = (µ, σ 2 ) | σ 2 > ε}. Consider only predictions f ∈ Pεn . Then, the NLL is decision-aligned w.r.t. {Uλ }λ∈R>ε under π(λ) = ελ−2 and hy (m) = 2εm − ε log(2πε)

∀y.

Proof. Restricting to λ > ε and fi ∈ Pε for all i, the negative expected pointwise utility is Z σi2 Z ∞ λπ(λ)dλ + (µi − yi )2 π(λ)dλ. σi2

ε

Choose π(λ) as the probability density function of a Pareto(1, ε) distribution, given by π(λ) = ελ−2 for λ ≥ ε. Substituting this density into the integral gives Z σi2 Z ∞
(µi − yi )2 1 1 dλ = ε log σi2 − log ε + ε . ε dλ + ε(µi − yi )2 2 λ σi2 ε σi2 λ By linearity of expectation, the negative expected dataset-level utility is the average of the pointwise expected risks:  Z n  1X (µi − yi )2 2 ε log σi + ε −Uλ,y (f )π(λ)dλ = − ε log ε n i=1 σi2 Θ " n  # (µi − yi )2 1X 1 2 log σi + − ε log ε, (15) = 2ε n i=1 2 2σi2 which closely resembles the NLL. We can express (15) as an affine transformation of the NLL: Z −Uλ,y (f )π(λ)dλ = 2ε · NLLy (f ) − ε log(2πε). Θ

Defining hy (x) = 2εx − ε log(2πε) for all y, which is strictly increasing for ε > 0, we satisfy (1). Thus, M = NLL is decision-aligned with the selective prediction utility family under π(λ) = λε2 and hy (m) = 2εm − ε log(2πε) for all y. C.6

Proof of Proposition 3.14

Proposition 3.14. The NLL, MSE, ECE, MCE, R-AUC, and E-Det are not decision-aligned w.r.t. {Uϕ }ϕ∈Φ . Proof. For any M ∈ {NLL, MSE, ECE, MCE, R-AUC, E-Det}, assume for contradiction that M is decision-aligned w.r.t. {Uϕ }ϕ∈Φ , i.e., that there exists a prior π on Φ s.t. for any fixed labels y there exists a strictly increasing function hy : Im(My ) → R with Z
hy My (f ) = −Uϕ,y (f )π(ϕ) dϕ ∀f . Φ

Fix any strictly increasing sequence of variances 0 < v1 < v2 < · · · < vn and define the label vector y = 0 = (0, . . . , 0). Consider two prediction vectors with identical variances but different means:

f 1 := (0, . . . , 0), (v1 , . . . , vn ) , f 2 := (1, . . . , 1), (v1 , . . . , vn ) . For any ϕ = (k, γ) ∈ Φ and any f = (µ, σ 2 ), the selection rule ranks instances by the score µi − γσi2 . Since both f 1 and f 2 have constant means and the same strictly ordered variances, for every ϕ = (k, γ) the ordering of the scores is the ordering of −γvi , hence Sϕ (f 1 ) = Sϕ (f 2 ) = {1, . . . , k}

∀(k, γ) ∈ Φ.

Therefore, for every ϕ = (k, γ), k

1X (0 − γvi ) = Uϕ,0 (f 2 ). Uϕ,0 (f 1 ) = k i=1 20

Consequently, the expected utility under any prior π coincides: Z Z −Uϕ,0 (f 1 )π(ϕ)dϕ = −Uϕ,0 (f 2 )π(ϕ)dϕ. Φ

Φ

Since h0 is injective, decision-alignment of M would require that M0 (f 1 ) = M0 (f 2 ). We now show that this fails. Case M = MSE: Since y = 0, MSE0 (f 1 ) = 0 Case M = NLL: For Gaussian NLL, n 1X1 log(2πvi ) NLL0 (f 1 ) = n i=1 2

̸=

1 = MSE0 (f 2 ). n

̸=

1X n i=1



1 1 2 log(2πvi ) + 2v i

 = NLL0 (f 2 ).

Case M ∈ {ECE, MCE}: For y = 0 and f 1 = ((0, . . . , 0), (v1 , . . . , vn )), each observation equals the predictive mean, level αj and every index i we have √ so for every miscoverage √ yi = µi ∈ [µi − z1−αj /2 vi , µi + z1−αj /2 vi ], hence α̂j (f 1 , 0) = 0. In contrast, for f 2 = ((1, . . . , 1), (v1 , . . . , vn )) we have |yi − µi |√= 1 for all i, and therefore the empirical miscoverage α̂j (f 2 , 0) depends on whether 1 ≤ z1−αj /2 vi holds. Choose the variance sequence s.t. v1 is √ small enough s.t. 1 > z1−α∗ /2 v1 for at least one level α∗ and the remaining vi are large enough √ s.t. 1 ≤ z1−αj /2 vi for all αj . Then, α̂∗ (f 2 , 0) = n1 ̸= 0 = α̂∗ (f 1 , 0), and all other empirical miscoverages are 0 for both predictions, hence, ECE0 (f 1 ) ̸= ECE0 (f 2 )

and

MCE0 (f 1 ) ̸= MCE0 (f 2 ).

Case M = R-AUC: In regression, R-AUC is computed from the retention sets Rr obtained by sorting instances by Gaussian predictive entropy (which is a strictly increasing function of σi2 ), and uses the MSE on the retained subset as the error rate. With our choice of f 1 , f 2 , the retained sets Rr = {π(r + 1), . . . , π(n)} are identical under f 1 , f 2 (because σ 2 is the same in both). Moreover, on any nonempty retained set Rr we have MSE0 (f 1,i:i∈Rr ) = 0

̸=

1 = MSE0 (f 2,i:i∈Rr ).

Consequently, R-AUC0 (f 1 ) ̸= R-AUC0 (f 2 ). Case M = E-Det: In regression, we use the Gaussian predictive entropy as uncertainty score si = s(σi2 ) (strictly increasing in σi2 ) and define error indicators by ei := 1 yi −µi >τ for fixed ε > 0 |yi |+ε

and τ > 0. Choose a = 0 and b = (τ + 1)ε. Next, pick variances 0 < v1 < · · · < vn and set σ 2 = (v1 , vn , . . . , vn ), so that s(v1 ) < s(vn ). Define two forecasts with identical variances but swapped mean offsets, f 1 = (µ1 , σ 2 ), f 2 = (µ2 , σ 2 ), where µ1 = ( a, b, b, . . . , b ),

µ2 = ( b, a, a, . . . , a ).

Note that the corresponding expected utilities still match, even for a random tie-breaking rule, since the utility only depends on 0 and σ 2 . Under f 1 we have e1 = 0 and ei = 1 for i ≥ 2, and the unique point with e = 0 has uncertainty score s(v1 ) while every point with e = 1 has uncertainty score s(vn ). Since s(v1 ) < s(vn ), all positives have strictly larger scores than all negatives, hence E-Det0 (f 1 ) = −1. Under f 2 we have e1 = 1 and ei = 0 for i ≥ 2, so the unique positive has score s(v1 ) while all negatives have score s(vn ), implying E-Det0 (f 2 ) = 0. In all cases, we obtain a contradiction. Therefore no such prior π can exist, and M is not decisionaligned w.r.t. {Uϕ }ϕ∈Φ .

D

Relationship between decision-alignment and properness

In this section, we prove that many decision-aligned metrics (those where the transformations hy are affine) are proper. Note, however, that the converse is not true. While any PSR M is decision-aligned to some decision-family, which we can simply define as Uθ = θM (and considering π(θ) = δ1 (θ)), a PSR is not decision-aligned to every utility family. We discuss several such examples in this paper (for example, the NLL is not decision-aligned w.r.t. the top-k utility family). 21

Proposition D.1 (Decision-alignment with affine hy ⇒ properness). Suppose that the metRric M is decision-aligned w.r.t. {Uθ }θ∈Θ under π with transformations hy , and assume E [|Uθ,Y (f )|] π(θ)dθ < ∞ for all (f , y). Further, assume there exists a constant a > 0 and Θ Y ∼p a function b : Y n → R s.t. hy (m) = am + b(y) for all y. Then, M is a PSR. If, in addition, the Bayes acts for each Uθ are unique under all f , then M is strictly proper. Proof. Define S(f , y) := EUθ∼π,y (f ). By Dawid and Musio [13], we know that −Uθ,y (f ) is a PSR for all θ. We first show that this implies that S(f , y) is a PSR as well. Denote by p the true distribution on Y n . By definition of properness, it holds for all predictions f that EY ∼p [−Uθ,Y (p)] ≤ EY ∼p [−Uθ,Y (f )]

for all θ ∈ Θ.

Multiplying by the positive π(θ) and integrating both sides w.r.t. θ gives Z Z EY ∼p [−Uθ,Y (p)] π(θ)dθ ≤ EY ∼p [−Uθ,Y (f )] π(θ)dθ, Θ

Θ

where the integrals exist by assumption. By Fubini-Tonelli (again using the assumed finiteness of expectations), Z  Z EY ∼p [−Uθ,Y (f )] π(θ)dθ = EY ∼p −Uθ,Y (f )π(θ)dθ = EY ∼p [S(f , Y )] . Θ

Θ

With the same equality on p instead of f , we get EY ∼p [S(p, Y )] ≤ EY ∼p [S(f , Y )] , which shows that S(f , y) is itself a PSR. Now, since aM (f , y) + b(y) = EUθ∼π,y ⇔ M (f , y) =

1 1 EUθ∼π,y − b(y), a a

we have for any prediction f , 1 1 EY ∼p [S(f , Y )] − EY ∼p [b(Y )], a a so the set of minimizers of EY ∼p [M (f , Y )] coincides with that of EY ∼p [S(f , Y )]. Thus, M is proper. If finally the Bayes acts for Uθ are unique under all f , −Uθ,y (f ) is a strict PSR, all inequalities become strict inequalities, and M will be a strict PSR. EY ∼p [M (f , Y )] =

E

Counterexample for the converse relation of Lemma 3.6

Consider P = Y = R. Define the pointwise-separable (and with this also coordinate-independent) metric n 1X (fi − yi )2 . M (f , y) := n i=1 Let Θ = {−1, +1} and fix the prior π(+1) = π(−1) = 21 . Define the non-separable functional h(f , y) := 1f1 >f2 , which depends on the joint ordering of (f1 , f2 ) and hence is not pointwiseseparable. Now define a utility family {Uθ }θ∈Θ by U+1 (f , y) := −M (f , y) − h(f , y),

U−1 (f , y) := −M (f , y) + h(f , y).

Then both U+1 and U−1 are non-separable (because of the ±h term), yet Z
1
1 −Uθ (f , y)dπ(θ) = −U+1 (f , y) + −U−1 (f , y) 2 2 Θ
1
1 = M (f , y) + h(f , y) + M (f , y) − h(f , y) 2 2 = M (f , y). Hence, M is decision-aligned w.r.t. the family {Uθ }θ∈Θ under π, even though every Uθ in the family is non-separable. 22

F

Omitted metric definitions

In this section, we define all metrics used in this paper and prove for each metric whether it is coordinate-independent or not. We start by defining each metric and giving a short intuition of why they are coordinate-(in)dependent. At the bottom of this section, we state a proposition, where we formally prove these intuitions. Negative log-likelihood (NLL)

For binary classification, the pointwise NLL is defined as

lNLL (f, y) = −y log(f ) − (1 − y) log(1 − f ), and for regression (f = (µ, σ 2 )), we use the Gaussian NLL, 2

lNLL (f, y) = 12 log(2πσ 2 ) + (y−µ) 2σ 2 . The dataset-level loss is defined as the average, n

NLL(f , y) =

1X lNLL (fi , yi ). n i=1

Thus, by definition, the NLL is pointwise-separable, hence also coordinate-independent. Further, the NLL is a PSR [9]. Brier score (BS)

The pointwise BS is defined as lBS (f, y) = (f − y)2 ,

and the dataset-level loss is defined as the average, n

BS(f , y) =

1X lBS (fi , yi ). n i=1

Thus, by definition, the BS is pointwise-separable, hence also coordinate-independent. Further, the BS is a PSR [9]. Accuracy (Acc) Accuracy (Acc) is conventionally defined as the ratio of correctly classified instances when using a generic decision threshold of 0.5. To conform with our notion that smaller metrics correspond to better performance, we consider the negative conventional accuracy as our Acc. With this, it is the sample average of the pointwise decision 0-1 loss, the indicator function of a false classification of any kind, minus 1, i.e., l0-1 (f, y) = y1f ≤0.5 + (1 − y)1f >0.5 , n 1X Acc(p, y) = l0-1 (pi , yi ) − 1. n i=1 Thus, by definition, Acc is pointwise-separable, hence also coordinate-independent, however, the Acc is not a PSR [9]. Mean squared error (MSE) The pointwise mean squared error (MSE) is the squared-error loss, defined as lMSE (f, y) := (µ − y)2 , where f = (µ, σ 2 ), and the dataset MSE is n

MSE(f , y) :=

1X lMSE (fi , yi ). n i=1

Thus, by definition, the MSE is pointwise-separable, hence also coordinate-independent. The MSE only evaluates the predictive mean, not the full Gaussian predictive distribution, and can therefore not be considered proper in terms of the full probabilistic report f ∈ P. 23

Expected calibration error (ECE) A widely used empirical calibration metric in binary classification is the ECE, which measures the discrepancy between empirical frequencies and stated probabilities (aka calibration) within bins of the prediction space. Formally, fix a binning scheme B = {B1 , . . . , Bm } that partitions [0, 1]. The ECE is defined as ECE(f , y) :=

m X nj j=1

n

freq(Bj ) − conf(Bj ) ,

P where nj is the number of indices i s.t. fi ∈ Bj , freq(Bj ) = n1j i: fi ∈Bj yi , and conf(Bj ) = P 1 i: fi ∈Bj fi . We use 10 equally spaced bins. The ECE is not a PSR [6]. The ECE is coordinatenj dependent, since it couples coordinates through bin membership and bin statistics. To use the ECE in regression, we follow Cui et al. [42], who assess calibration of regression models through nominal coverage levels α1 , . . . , αm ∈ (0, 1) and the corresponding Gaussian prediction intervals. We use α ∈ {0.5, 0.8, 0.9, 0.95}. Let α̂j ((µ, σ 2 ), y) denote the empirical miscoverage of the (1 − αj )-level prediction interval. Then, the empirical regression ECE is defined as m

ECE((µ, σ 2 ), y) :=

1 X α̂j ((µ, σ 2 ), y) − αj . m j=1

As in the binary classification case, the regression ECE is not a PSR and coordinate-dependent. Maximum calibration error (MCE) The MCE measures the worst-case deviation between empirical frequencies and stated probabilities across bins of the prediction space. Using the same notation as for ECE, the MCE is defined as MCE(f , y) :=

max j∈{1,...,m}

freq(Bj ) − conf(Bj ) .

Like the ECE, the MCE is not a PSR and MCE is coordinate-dependent. We adapt the MCE to regression in the same way as the ECE. Area under the retention curve (R-AUC) The R-AUC [43] measures the usefulness of a model’s uncertainty estimates for selecting correct predictions. For varying certainty thresholds τ , the retention curve tracks the error rate of a model when only predicting on the the subset with certainty at least τ . When uncertainty is informative, incorrect predictions tend to be rejected first, resulting in a steep decline of the retention curve. By integrating this curve, R-AUC yields a single scalar that reflects both overall predictive accuracy and the ability of uncertainty estimates to rank errors. Formally, let si = s(fi ) denote a scalar uncertainty score for instance i, with larger values indicating higher uncertainty. We use the predictive entropy, si = −fi log fi − (1 − fi ) log(1 − fi ). Let π be a permutation of {1, . . . , n} s.t. sπ(1) ≥ · · · ≥ sπ(n) . For r ∈ {0, 1, . . . , n}, define the retained index set Rr := {π(r + 1), . . . , π(n)}, corresponding to retaining a fraction τ = |Rr |/n of predictions and deferring the remaining r most uncertain instances to an oracle. The area under the retention curve (R-AUC) is then defined as n−1

R-AUC(f , y) :=

1X 1 + Acc(f i:i∈Rr , y i:i∈Rr ) , n r=0

where 1 + Acc is the error rate—remember that our Acc is the negative conventional accuracy. The R-AUC is not a PSR, since it can be improved by pushing uncertainty scores for likely-wrong predictions artificially high, so honest reporting is not necessarily optimal. Further, the R-AUC is coordinate-dependent: changing a single prediction fi affects not only the error term 1fi ̸=yi associated with instance i, but also the ordering induced by the uncertainty scores {sk }nk=1 . For regression models, we use the Gaussian predictive entropy as uncertainty score and the MSE as error rate. Like in binary classification, the regression R-AUC is not a PSR and is coordinatedependent. 24

Error detection (E-Det) E-Det [44] evaluates how well uncertainty can be used as a score to discriminate between correct and incorrect predictions. We define the hard predictions as ŷi = 1fi >0.5 and the corresponding error indicators as ei = 1ŷi ̸=yi . Again writing si for the uncertainty score (where we use predictive entropy, as above), E-Det is defined as the classical area under the receiver operating curve (AUROC), using si as predictions for the labels ei . To conform with our notion that smaller metrics correspond to better performance, we consider the negative conventional error detection as our E-Det. The E-Det is not a PSR and the E-Det is coordinate-dependent because of its ranking characteristic, similar to the R-AUC. In regression, we again use the Gaussian predictive entropy as uncertainty score and define errors ei y−µ as predictions whose relative errors | |y|+ε | exceed 10% (using a small ε > 0 to ensure the relative error is well-defined). Proposition F.1 (Coordinate-(in)dependence of common UQ evaluation metrics). The NLL, BS, Acc, and MSE are coordinate-independent, and the ECE, MCE, R-AUC, and E-Det are coordinatedependent. Proof. The coordinate-independence of the NLL, BS, Acc, and MSE follows directly from their pointwise-separability by definition. We now prove coordinate-dependence of the remaining metrics. ECE & MCE: We prove the claim for ECE; the claim for MCE then follows by the same construction (indeed, in our construction, only one bin is populated, so ECE = MCE). Step 1: choose a rational frequency inside a non-degenerate bin. Let B = (ℓ, u] be a bin of nonzero width, i.e., u − ℓ > 0. Pick any rational number q ∈ (ℓ, u); write it as q = k/t with integers t ≥ 2 and k ∈ {0, 1, . . . , t}. (Existence is guaranteed since every non-degenerate interval contains rationals.) Set n := t and fix labels y with exactly k ones and t − k zeros. Hence, for any prediction vector f for which all coordinates lie in B, we have t

freq(B) =

1X k yi = = q. t i=1 t

Step 2: fix two values a < b in the same bin. Choose any a, b ∈ B with a < b and with a and b sufficiently close to q so that the values defined below remain inside B. (Concretely: since B is an open neighborhood of q relative to (ℓ, u], there exists δ > 0 s.t. (q − δ, q + δ) ⊆ B; choose a, b ∈ (q − δ, q + δ) with a < b.) Fix an index i ∈ {1, . . . , t} (any choice works). Step 3: build two backgrounds that make the direction of change flip. Define c := tq−a t−1 and tq−b ′ c := t−1 . By construction, if we set all coordinates of f except i equal to c (respectively c′ ), then the bin confidence equals q when fi = a (respectively fi = b): X  a + (t − 1)c X  b + (t − 1)c′ 1 a + tq − a 1 = = q, = q. a+ c = b+ c′ = t t t t t r̸=i

r̸=i

Since a and b were chosen sufficiently close to q, the values c and c′ are also close to q and thus lie in q−b ′ B as well (note that we can rewrite c, c′ as perturbations of q: c = q + q−a t−1 , c = q + t−1 . Therefore, in both constructions, all predictions fall into the same bin B. Now define two prediction vectors: f 1 := (a, c, . . . , c), | {z }

f̃ 1 := (b, c, . . . , c), | {z }

f 2 := (a, c′ , . . . , c′ ), | {z }

f̃ 2 := (b, c′ , . . . , c′ ), | {z }

t−1 times

and

t−1 times

t−1 times

t−1 times

where the distinguished coordinate is i (we suppress the explicit placement notation for readability). Since all coordinates lie in B, the ECE reduces to the single-bin absolute deviation: ECEy (f ) = freq(B) − conf(B) . For f 1 we have conf(B) = q = freq(B), hence ECEy (f 1 ) = 0. But for f̃ 1 we have conf(B) =

b + (t − 1)c b + (tq − a) b−a = =q+ , t t t 25

so

 b−a b − a = > 0. ECEy (f̃ 1 ) = q − q + t t Thus, under
the background
r = (c, . . . , c), changing fi from a to b increases the ECE: ECEy (a, r) < ECEy (b, r) . Conversely, for f̃ 2 we again have perfect calibration in B by construction, hence ECEy (f̃ 2 ) = 0, whereas for f 2 we compute conf(B) =

a + (t − 1)c′ a + (tq − b) b−a = =q− , t t t

so

 b−a b − a = > 0. ECEy (f 2 ) = q − q − t t Hence, under
the background
r ′ = (c′ , . . . , c′ ), changing fi from a to b decreases the ECE: ECEy (a, r ′ ) > ECEy (b, r ′ ) .

Transfer to regression. Fix a nominal level α ∈ (0, 1). For Gaussian predictive distributions, let n

α̂((µ, σ 2 ), y) :=

1X 1 / α (µi ,σi2 )} n i=1 {yi ∈I

denote the empirical miscoverage of the (1 − α)-level prediction interval Iα (µi , σi2 ). Since the regression ECE (resp. MCE) is obtained by averaging (resp. maximizing) the absolute deviations |α̂ − α| over the finitely many levels αj , coordinate-independence would in particular require coordinate-independence of the contribution of each fixed level α (as we can easily pick labels and predictions such that slightly perturbing a prediction only affects one miscoverage term). We therefore fix α and construct a counterexample for |α̂ − α|. As in the classification proof, fix an index i and two alternative predictions a and b for the ith coordinate (i.e., two Gaussian reports (µi , σi2 )), such that the ith prediction interval covers yi under a but does not cover yi under b. For the remaining coordinates k ̸= i, construct two background prediction vectors r and r ′ as follows: in r, choose the (µk , σk2 ) so that exactly αt of the t − 1 background prediction intervals fail to cover their labels when the ith coordinate is set to a, whereas in r ′ , choose them so that exactly αt background intervals fail to cover their labels when the ith coordinate is set to b (such constructions are always possible by choosing the remaining intervals sufficiently wide or narrow.). By construction, we then have α̂((a, r), y) = α,

α̂((b, r ′ ), y) = α,

so that |α̂ − α| = 0 in both cases. However, switching a to b under the background r increases the number of miscovered instances by one and thus increases |α̂ − α|, whereas switching a to b under the background r ′ decreases the number of miscovered instances by one and thus decreases |α̂ − α|. Hence, |α̂((a, r), y) − α| < |α̂((b, r), y) − α|,

|α̂((a, r ′ ), y) − α| > |α̂((b, r ′ ), y) − α|.

This flip of inequalities shows that the direction of change of the calibration error induced by modifying a single coordinate depends on the remaining coordinates. Therefore, the regression ECE is coordinate-dependent. Moreover, by choosing a nominal level α whose absolute miscoverage deviation strictly dominates all others, the same construction ensures that this level uniquely attains the maximum in the definition of the regression MCE. Hence, the same argument applies to the regression MCE, which is therefore also coordinate-dependent. R-AUC: We give explicit counterexamples that violate coordinate-independence. We start with the binary classification case. Let n = 2 and fix labels y = (1, 0). For a probability f ∈ (0, 1), write s(f ) := −f log f − (1 − f ) log(1 − f ) for the predictive entropy, and let the hard prediction be ŷ(p) := 1p>0.5 . For n = 2 the R-AUC reduces to  1 R-AUCy (f ) = err({1, 2}) + err(R1 ) , 2 where err(·) is the classification error rate, and R1 is obtained by rejecting the single most uncertain instance. Fix the index i = 1 and choose a := 0.9, b := 0.51. Note that ŷ(a) = ŷ(b) = 1, hence the first instance is correct under both a and b, but s(b) > s(a). 26

Background 1. Let the second prediction coordinates be r := 0.8. Then ŷ(r) = 1 ̸= y2 , so the second instance is wrong, and moreover s(b) > s(r) > s(a). Hence, with f = (a, r) the most uncertain instance is 2, so R1 = {1} and err(R1 ) = 0; while with f = (b, r) the most uncertain instance is 1, so R1 = {2} and err(R1 ) = 1. In both cases the full-set error rate is err({1, 2}) = 1/2 (exactly one error). Therefore,  1  11 3 11 R-AUCy ((a, r)) = +0 = < = + 1 = R-AUCy ((b, r)), 2 2 4 4 2 2 so changing f1 from a to b increases the R-AUC. Background 2. Let the second prediction coordinate be r′ := 0.01. Then ŷ(r′ ) = 0 = y2 , so the second instance is correct, and s(r′ ) < s(a) < s(b). Hence, regardless of whether f1 = a or f1 = b, the most uncertain instance is 1 and thus R1 = {2}, yielding err(R1 ) = 0. Moreover, both instances are correct under (a, r′ ) and (b, r′ ), so err({1, 2}) = 0 in both cases. Thus, R-AUCy ((a, r′ )) = R-AUCy ((b, r′ )) = 0. Combining the two backgrounds, we have exhibited fixed labels y, an index i = 1, and values a, b s.t. R-AUCy ((a, r)) < R-AUCy ((b, r)), while R-AUCy ((a, r′ )) = R-AUCy ((b, r′ )), violating coordinate-independence. Now we move on to regression. Let n = 2 and fix labels y = (0, 0). Reports are fi = (µi , σi2 ), the uncertainty score is the Gaussian predictive entropy s(µ, σ 2 ) := 12 log(2πe σ 2 ), which is strictly increasing in σ 2 , and the error rate on a retained set is the MSE computed from the predictive means. Fix i = 1 and choose the two alternative reports a := (µ1 , σ12 ) = (1, 0.12 ),

b := (µ1 , σ12 ) = (1, 102 ),

so that the mean-squared error of the first instance equals (µ1 − y1 )2 = 1 under both a and b, but the uncertainty score satisfies s(b) > s(a). Background 1. Let the remaining coordinate be r := (µ2 , σ22 ) = (0, 12 ), so the second instance has squared error (µ2 − y2 )2 = 0 and uncertainty score s(r) with s(a) < s(r) < s(b). Hence, with (a, r) the most uncertain instance is 2 and R1 = {1}, whereas with (b, r) the most uncertain instance is 1 and R1 = {2}. The full-set MSE equals 1+0 1 = 2 2 for both (a, r) and (b, r), while the retained-set MSE equals 1 under (a, r) and 0 under (b, r). Therefore,  3  11 1 11 R-AUCy ((a, r)) = +1 = > = + 0 = R-AUCy ((b, r)), 2 2 4 4 2 2 so changing f1 from a to b decreases the regression R-AUC. MSE({1, 2}) =

Background 2. Let the remaining coordinate be r′ := (µ2 , σ22 ) = (0, 0.012 ), so s(r′ ) < s(a) < s(b). Then the most uncertain instance is 1 under both (a, r′ ) and (b, r′ ), hence R1 = {2} in both cases. Again MSE({1, 2}) = 1/2, and the retained-set MSE is 0 in both cases, so 1 . 4 Thus, the strict comparison between R-AUCy ((a, ·)) and R-AUCy ((b, ·)) depends on the remaining coordinate, proving that the regression R-AUC is also coordinate-dependent. R-AUCy ((a, r′ )) = R-AUCy ((b, r′ )) =

E-Det: Since E-Det is the negative conventional AUROC, it suffices to show that the conventional E-Det is coordinate-dependent; multiplying by −1 preserves coordinate-(in)dependence. We start with binary classification. Let n = 3, fix labels y = (1, 0, 1), and let ŷ(f ) := 1f >0.5 . Define the error indicators ei = 1ŷ(fi )̸=yi , and let AUROC(s, e) be the standard AUROC treating larger si as stronger evidence for ei = 1. Fix i = 1 and choose two values a := 0.9, b := 0.49. Then ŷ(a) = 1 (hence e1 = 0 under a) while ŷ(b) = 0 (hence e1 = 1 under b), and moreover s(b) > s(a) since predictive entropy is maximized at 0.5. We construct two backgrounds r, r ′ ∈ (0, 1)2 (for coordinates 2, 3) that yield different directions of change. 27

Background 1. Set (f2 , f3 ) = (0.999, 0.6). Then ŷ(f2 ) = 1 ̸= y2 = 0, so e2 = 1, and ŷ(f3 ) = 1 = y3 , so e3 = 0. Moreover, s(0.999) < s(0.9) < s(0.6) < s(0.49). Hence, under (a, r) the positive set is {2} and the negative set is {1, 3}, and the unique positive score s2 = s(0.999) is smaller than both negative scores, so AUROCy ((a, r)) = 0. Under (b, r) the positive set is {1, 2} and the negative set is {3}, and we have s(b) > s(0.6) > s(0.999), hence exactly one of the two positive-negative comparisons is correct, so AUROCy ((b, r)) = 12 . Thus, changing f1 from a to b increases AUROC under background r. Background 2. Set (f2 , f3 ) = (0.51, 0.99). Then ŷ(f2 ) = 1 ̸= y2 = 0, so e2 = 1, and ŷ(f3 ) = 1 = y3 , so e3 = 0. Moreover, s(0.51) is close to maximal and therefore larger than both s(0.9) and s(0.99). Hence, under (a, r ′ ) the positive set is {2} and negatives are {1, 3}, and s2 exceeds both negative scores, so AUROCy ((a, r ′ )) = 1. Under (b, r ′ ) the positives are {1, 2} and the negative is {3}, and both positive scores s(b) and s(0.51) exceed the negative score s(0.99), so again AUROCy ((b, r ′ )) = 1. Thus, under background r ′ , the same change from a to b leaves AUROC unchanged, violating coordinate-independence. We now continue with the regression case. Let n = 3 and fix labels y = (1, 1, 1). Reports are fi = (µi , σi2 ). Fix τ > 0 and ε > 0, and define error indicators ei := 1n yi −µi ≥τ o . Define the |yi |+ε

conventional AUROC using scores si = s(fi ) to predict labels ei . Fix i = 1 and choose two reports a := (µ1 , σ12 ) = (1, 0.12 ),

b := (µ1 , σ12 ) = (1 + (|1| + ε)τ, 102 ).

Then under a we have e1 = 0, while under b we have e1 = 1 (by equality in the threshold), and s(b) > s(a). Background 1. Set f2 = (1 + (|1| + ε)τ, 0.012 ),

f3 = (1, 12 ).

Then e2 = 1 and e3 = 0. Moreover, by monotonicity in σ 2 , s(f2 ) < s(a) < s(f3 ) < s(b). Hence, under (a, r) the positive set is {2} and negatives are {1, 3}, and the unique positive score is smaller than both negative scores, giving AUROCy ((a, r)) = 0. Under (b, r) the positives are {1, 2} and the negative is {3}, and exactly one of the two positive-negative comparisons is correct (since s(b) > s(f3 ) > s(f2 )), giving AUROCy ((b, r)) = 12 . Thus, AUROC increases under background r when changing f1 from a to b. Background 2. Set f2 = (1 + (|1| + ε)τ, 102 ),

f3 = (1, 0.012 ).

Then e2 = 1 and e3 = 0, and s(f2 ) is large while s(f3 ) is small. Consequently, under (a, r ′ ) the positive set is {2} and negatives are {1, 3}, and the positive score exceeds both negative scores, so AUROCy ((a, r ′ )) = 1. Under (b, r ′ ) the positives are {1, 2} and the negative is {3}, and both positive scores exceed the negative score, so again AUROCy ((b, r ′ )) = 1. Thus, under background r ′ , the same change from a to b leaves AUROC unchanged, violating coordinate-independence.

G

Generalization to the multiclass and multivariate setting

The purpose of this appendix is to demonstrate that our framework extends beyond the binary and univariate setting. We perform the analysis on three metrics and one utility each. Analogous results for additional metrics and decision families can be derived using the same template. G.1

Multiclass classification

We consider probabilistic K-class classification models, K > 2, f : X → ∆K−1 , mapping from PK the input space X to the (K−1)-simplex ∆K−1 := {p ∈ [0, 1]K : j=1 pj = 1}, so the prediction for an instance is f = (f 1 , . . . , f K ) with f j denoting the predicted probability that the true label y ∈ Y := [K] := {1, . . . , K} equals j. As evaluation metrics, we consider the multiclass NLL, BS, and ECE. We define n

NLL(f , y) =

1X − log fiyi , n i=1

n

BS(f , y) =

28

K

2 1 XX j fi − 1yi =j . n i=1 j=1

The multiclass ECE uses the standard extension to vector-valued predictions via one-versus-all binning (see, e.g., [45]). The NLL and BS are pointwise-separable (hence coordinate-independent) by definition. The multiclass ECE is coordinate-dependent by the same bin-coupling argument as in the binary case (Proposition F.1). As downstream task, we generalize the binary decision from Section 3.2.1 to multiclass via a one-vsrest decomposition. For each class j ∈ [K], we consider the binary decision “flag instance as class j or not,” with cost threshold c ∈ (0, 1). Defining the binary label ỹ j := 1y=j , the Bayes act is to flag class j whenever f j > c. The pointwise utility of a prediction f with true label y for class j at threshold c is uj,c (f, y) = −ỹ j (1 − c)1f j ≤c − (1 − ỹ j )c1f j >c . This is exactly the binary utility uc from Section 3.2.1, applied to the binary prediction–label pair (f j , ỹ j ). The combinedPdecision family is indexed by θ = (j, c) ∈ [K] × (0, 1), with dataset-level n utility Uj,c (f , y) = n1 i=1 uj,c (fi , yi ). In particular, Uj,c is pointwise-separable. Proposition G.1. The multiclass NLL and BS are decision-aligned w.r.t. {Uj,c }j∈[K],c∈(0,1) with πNLL (j, c) = 1/c and πBS (j, c) = 2, and hy = id for all y. Proof. NLL: Evaluating the per-instance expected negative utility under π(j, c) = 1/c gives K Z 1 X j=1

−uj,c (fi , yi )

0

1 dc = c

Z 1

X 1−c dc + y c fi i

j̸=yi

Z fij 1 dc 0

  X j = − log fiyi − 1 + fiyi + fi j̸=yi

= − log fiyi − 1 + fiyi + (1 − fiyi ) = − log fiyi . P In the last step we used the simplex constraint j̸=yi fij = 1 − fiyi . Averaging over instances, EUy (f ) = NLL(f , y), so the NLL is decision-aligned with hy = id. PK BS: The multiclass BS decomposes as BS(f , y) = j=1 BS(j) (f , y), where BS(j) (f , y) is the binary BS for the j-th one-vs-rest problem. By Proposition 3.7, each binary BS admits the integral PK R 1 representation (3) with prior π(c) = 2, giving BS(f , y) = j=1 0 −Uj,c (f , y) · 2 dc, which matches Definition 3.1 with hy = id.

As in the binary case, the NLL and BS implicit priors are pathological. The ECE is not decisionaligned as a direct consequence of Lemma 3.6. Corollary G.2. The multiclass ECE is not decision-aligned w.r.t. {Uj,c }j∈[K],c∈(0,1) . G.2

Multivariate regression

We consider probabilistic multivariate regression models with output dimension D > 1. We assume a Gaussian predictive distribution and consider models of the form f : X → RD × SD ++ , mapping from the input space X to the predictive mean µ ∈ RD and predictive covariance Σ ∈ SD ++ (the cone of D × D symmetric positive definite matrices), so f = (µ, Σ). The regression target is y ∈ Y := RD . As evaluation metrics, we consider the multivariate NLL, MSE, and the energy score (ES) [9].15 We define  n  1X D 1 1 ⊤ −1 NLL(f , y) = log(2π) + log det(Σi ) + (yi − µi ) Σi (yi − µi ) , n i=1 2 2 2 n

MSE(f , y) =

1X ∥yi − µi ∥2 . n i=1

15 To the best of our knowledge, there is no established notion of the ECE for multivariate regression. Therefore, we consider

the ES, which is a more established metric in this context.

29

For a predictive distribution f and observation y, the per-instance ES is defined as les (f, y) = EX∼f ∥X − y∥ − 12 EX,X ′ ∼f ∥X − X ′ ∥, where X, X ′ are independent draws from f and ∥·∥ denotes the Euclidean norm, and the dataset-level ES is the average. All three metrics are pointwise-separable by definition. As downstream task, we generalize selective prediction from Section 3.3.1 to multivariate output by applying the univariate selective-prediction rule coordinate-wise to each standardized output dimension and averaging over dimensions. Concretely, for each output dimension d ∈ [D] the decision-maker chooses an action in {predict, abstain}. To put dimensions of different scale on equal footing, costs are standardized by the marginal variance vd := Var(yd ), so the standardized squared-error loss for a point prediction in dimension d is (yd − µd )2 /vd , and the abstention cost is a dimensionless λ ∈ R>0 . Equivalently, this is the univariate selective-prediction problem of √ Section 3.3.1 applied independently to each of the standardized targets yd / vd .16 The Bayes act in dimension d is to predict µd whenever the standardized predictive variance Σdd /vd is at most λ. The pointwise utility of a prediction f = (µ, Σ) with true outcome y ∈ RD is  D  1 X (yd − µd )2 1Σdd ≤λvd + λ 1Σdd >λvd . uλ (f, y) = − D vd d=1

The dataset-level utility Uλ (f , y) is defined as the average pointwise utility. In particular, Uλ is pointwise-separable. Proposition G.3. The multivariate NLL, MSE, and ES are not decision-aligned w.r.t. {Uλ }λ∈R>0 . Proof. Write ri,d := Σi,dd /vd for the standardized marginal predictive variance and ei,d := (yi,d − µi,d )2 /vd for the standardized squared error. Consider any prior π. The per-instance expected negative utility takes the form s(fi , yi ) =

D  1 X ei,d S(ri,d ) + G(ri,d ) , D

(16)

d=1

R∞ Rt where S(t) := t π(λ) dλ and G(t) := 0 λ π(λ) dλ. In particular, s(fi , yi ) depends on Σi only through its (standardized) diagonal (ri,1 , . . . , ri,D ). We now show, for each metric, that a strictly monotone link to the expected utility cannot exist. MSE: The MSE does not depend on Σi at all. Fix labels y and set µi = yi for all i, so MSE(f P , y) = 0 1 regardless of the covariances. Then ei,d = 0 for all i, d, and (16) reduces to s(fi , yi ) = D d G(ri,d ). Rb Since π is nonzero, there exists an interval (a, b) ⊂ R>0 with a π(λ) dλ > 0, and for t1 , t2 ∈ (a, b) P ⊤ with t1 < t2 and t1 , t2 ̸= 1 we have G(t2 ) > G(t1 ). Setting Σi = tv1 e1 e⊤ 1 + d≥2 vd ed ed (where ed is the d-th standard basis vector), we obtain ri,1 = t and ri,d = 1 for d ≥ 2. Choosing t ∈ {t1 , t2 } yields two prediction vectors with identical MSE (both zero) but different expected utilities, contradicting the existence of a strictly increasing hy . For the NLL and ES, we use the following shared construction. Since all three quantities (NLL, ES, and expected utility) are pointwise-separable, fixing n = 2, any labels y, and f2 arbitrarily, the strictly increasing function hy would establish a strictly monotone link between the per-instance metric l(f1 , y1 ) and the per-instance expected utility s(f1 , y1 ). In particular, l(f1 , y1 ) ̸= l(f1′ , y1 ) must imply s(f1 , y1 ) ̸= s(f1′ , y1 ). Fix µ1 = y1 and consider two covariance matrices that share the same diagonal but differ in their off-diagonal structure:
√ ⊤ ΣA = diag(v1 , v2 , . . . , vD ), ΣB = ΣA + ρ v1 v2 e1 e⊤ 2 + e 2 e1 , for some ρ ∈ (0, 1) and v1 ̸= v2 where ed denotes the d-th standard basis vector. Both matrices are symmetric positive definite (the leading 2 × 2 block of ΣB has determinant v1 v2 (1 − ρ2 ) > 0). They share the same diagonal entries, hence the same standardized diagonal (1, 1,P . . . , 1), and since 1 µ1 = y1 also e1,d = 0 for all d. By (16), s((y1 , ΣA ), y1 ) = s((y1 , ΣB ), y1 ) = D d G(1). 16 Note that we apply the same scaling in the univariate case (see Appendix I), but we omitted this notation in the main text

for brevity.

30

1 NLL: Since µ1 = y1 , the per-instance NLL reduces to D 2 log(2π) + 2 log det(Σ1 ). We have Q Q 2 det(ΣA ) = d vd and det(ΣB ) = v1 v2 (1 − ρ ) d≥3 vd < det(ΣA ), so the NLL takes different values for ΣA and ΣB while the expected utility is the same. This contradicts the strictly monotone link.

ES: Since µ1 = y1 , the predictive distribution is N (y1 , Σ1 ) and for W := X − y1 , √
les (y1 , Σ1 ), y1 = (1 − 1/ 2) EW ∼N (0,Σ1 ) ∥W ∥. d PD 2 Writing η1 , . . . , ηD for the eigenvalues of Σ1 , by its eigendecomposition ∥W ∥2 = d=1 ηd Zd where qP Z1 , . . . , ZD are independent standard normal variables. The map Φ = (η1 , . . . , ηD ) 7→ D E ηd Zd2 is strictly concave on RD >0 (as the expectation of the strictly concave map η 7→ pP d=1 2 is concave for every realization z as the composition of the concave, nond ηd zd , which √ decreasing function · with a linear function of η). The eigenvalues of qΣA are (v1 , v2 , . . . , vD ), while
v1 +v2 v1 +v2 v1 −v2 2 2 2 those of ΣB are + ∆, 2 − ∆, v3 , . . . , vD with ∆ := ( v1 −v 2 2 ) + ρ v1 v2 > | 2 |. Hence ΣB ’s eigenvalues are a strict mean-preserving spread of ΣA ’s within the first two coordinates. By symmetry of Φ in its first two arguments, Φ(a) = Φ(a′ ) where a′ := (v2 , v1 , v3 , . . . , vD ), and similarly Φ(b) = Φ(b′ ) where b′ swaps the first two entries of b. Both pairs share the same midpoint ′ 2 v1 +v2 ā := 12 (a + a′ ) = 12 (b + b′ ) = ( v1 +v 2 , 2 , v3 , . . . , vD ). Since v1 ̸= v2 implies a ̸= a , and ′ ∆ > 0 implies b ̸= b , strict concavity of Φ gives Φ(ā) > Φ(a) and Φ(ā) > Φ(b). Moreover, a and b 2 lie on the line through ā in the direction (1, −1, 0, . . . , 0), with a at distance | v1 −v 2 | from ā and b at v1 −v2 v1 −v2 distance ∆ > | 2 |, so a = (1 − t)ā + tb for t := | 2 |/∆ ∈ (0, 1) (up to swapping b and b′ , which by symmetry does not affect Φ). By strict concavity of Φ along this segment,

Φ(a) > (1 − t)Φ(ā) + tΦ(b) > (1 − t)Φ(b) + tΦ(b) = Φ(b), where the second inequality uses Φ(ā) > Φ(b). Hence EW ∼N (0,ΣA ) ∥W ∥ = Φ(a) > Φ(b) = EW ∼N (0,ΣB ) ∥W ∥, so the ES takes different values for ΣA and ΣB while the expected utility is the same. This contradicts the strictly monotone link.

Binary decision

Selective prediction

π(θ)

10 5 0

0.0

0.2

0.4

0.6

0.8

utility parameter θ = c

πNLL

πBS

1.0

0

0.2

0.4

0.6

0.8

utility parameter θ = λ

πAcc

πNLL

∞

πMSE

Figure G.1: Illustrations of the pathological priors identified in Section 3 for the binary decision (left) and selective prediction (right) utility. The arrows indicate where the curves diverge.

H

PWU documentation

We now document our PWU metrics similarly to the model card framework [31]. Metric details The PWU metrics are a family of proper scoring rules for evaluating probabilistic predictions, introduced in this paper (Section 4). A PWU metric Mπ is parameterized by a deciR sion family {Uθ }θ∈Θ and a prior π over Θ, and is defined as Mπ (f, y) := Θ −Uθ (f, y) π(θ) dθ (Equation (4)). The construction applies to any decision family with finite expected utility under π. We instantiate four PWU metrics in the binary and univariate setting: Mπc (binary decisions), Mπλ 31

(selective prediction), Mπk (top-k selection in classification), and Mπϕ (top-k selection in regression with risk aversion). We further introduce two analogous PWU instantiations for the multiclass and multivariate setting in Appendix G. Inputs and outputs A PWU metric takes a first-order probabilistic prediction f and a label y, and returns a real-valued score (lower is better). Intended use PWU metrics are intended for general-purpose uncertainty quantification benchmarking when no specific downstream task is given. They are designed as a complement to, not a replacement for, conventional UQ metrics: each PWU encodes a single decision family, so we recommend reporting a set of PWUs alongside standard scoring rules to cover a range of plausible downstream scenarios. Prior elicitation The prior π controls which regions of the decision space the metric emphasizes. We justify our specific prior choices and provide general elicitation guidance in Appendix I; sensitivity to prior misspecification is analyzed in Appendix J.3, where we find that PWU metrics retain robust positive utility alignment even under extreme prior perturbations. Limitations No single PWU metric covers all downstream objectives, therefore, several PWU metrics are required to broadly represent a model’s decision-making ability. Prior elicitation introduces a modeling step that generic metrics avoid. Our PWUs’ priors may be misspecified in certain decision problems, which may degrade performance (even though we observed in Appendix J.3 that PWU metrics exhibit robust positive alignment even under severe prior misspecification). The scope of PWU metrics is currently restricted to first-order probabilistic predictions (Appendix A); extension to second-order UQ evaluation is left for future work. Computationally, evaluating Mπ requires integrating the negative utility against π, which requires numerical integration for prior choices that do not admit a closed-form Mπ . Mathematically, only utility–prior pairs are admitted that allow for a finite expected utility and hence a well-defined PWU. Out-of-scope use PWU metrics are not intended as training objectives—some are non-separable, making mini-batching ill-defined, some are non-differentiable (top-k selection, thresholding), and the integral adds computational cost. They are also not intended as a fairness or robustness diagnostic: alignment with a chosen utility says nothing about performance on subpopulations not captured by that utility. Broader impact Our work proposes evaluation metrics rather than new models, datasets, or deployment systems, so its societal impact is indirect and mediated by how UQ benchmarks influence model selection. We see two main directions of impact. On the positive side, PWU metrics encourage UQ benchmarking that is explicit about the downstream decision context and the assumed distribution over decision parameters. Standard UQ metrics encode implicit and often pathological priors over downstream costs; making these priors explicit can lead to model rankings that better track realized utility in deployed systems—particularly in cost-sensitive settings such as medical triage, credit approval, and selective prediction, where mismatches between the evaluation metric and the actual decision cost can have material consequences. On the negative side, PWU metrics depend on a chosen prior, and a poorly elicited or strategically chosen prior can shift model rankings in ways that favor certain models without that choice being apparent to downstream users. While our sensitivity analysis (Appendix J.3) shows that PWU metrics remain robust to substantial prior misspecification, transparent reporting of the prior used—ideally alongside conventional metrics and a range of plausible alternative priors, as we recommend in Section 4—is important to prevent the metric from being used to justify a predetermined conclusion. PWU metrics also do not address fairness or distributional concerns: alignment with a chosen utility says nothing about performance on subpopulations not captured by that utility, and a benchmark that selects models well on average may still select models that perform poorly on minority groups. We do not foresee a direct path from this work to malicious use: PWU metrics evaluate existing predictions and do not enable new capabilities such as generation, surveillance, or large-scale inference attacks. 32

I

Choosing PWU priors

Priors should be designed through consultation of the domain literature and expert knowledge. For each decision task, one should ask what range of decision parameters is plausible, and where the probability mass should lie. This process is neither straightforward nor unique—prior elicitation is a complex research area in it’s own right (see, e.g., [46]). Fortunately, PWU metrics are robust to misspecification; even PWUs corresponding to misspecified priors perform better than metrics corresponding to pathological priors, as demonstrated in our sensitivity analysis in Appendix J.3. In the remainder of this section, we provide concrete prior-elicitation guidance for benchmark designers who wish to construct their own PWU metrics. Appendix I.1 describes a four-step procedure for constructing a PWU metric in general, Appendix I.2 lists default prior families, Appendix I.3 provides a diagnostic prior-choice checklist, and Appendix I.4 documents the prior choices used in our experiments as worked examples of the procedure. I.1

A four-step PWU construction procedure

We recommend the following procedure for constructing a PWU prior. Step 1: Identify the decision families Specify the action sets for potential downstream use and the corresponding utility (or loss) structure. Express this as parameterized families {Uθ }θ∈Θ . Step 2: Identify the parameter spaces Θ Determine the units, range, and natural scale of θ. Is it bounded (cost ratios, selection fractions) or unbounded (abstention costs, risk-aversion coefficients)? For unbounded parameters, consider whether a natural rescaling exists (e.g., normalizing abstention costs by the empirical variance of y as we do in Section 3.3.1). Step 3: Specify prior families Select parametric families for πθ using the defaults in Appendix I.2. For most decision parameters that arise in practice, a Beta distribution on a bounded or rescaled domain suffices. Step 4: Set the prior’s mode and concentration Pin down the mode from a single domain anchor (e.g., “a typical false negative is roughly 10× as costly as a false positive” anchors c at ≈ 0.1). Set the concentration according to confidence in that anchor: a tight prior reflects strong belief in the modal value; a diffuse prior reflects broader uncertainty about the deployment context. We recommend verifying the choice using the diagnostics in Appendix I.3. I.2

Default prior families

Table I.1 summarizes default prior families for decision parameters that commonly arise in UQ benchmarking. Table I.1: Default prior families for common decision parameter types. For unbounded scale parameters, we recommend rescaling to a bounded domain (e.g., dividing by the empirical variance of y) and using a Beta prior, as we do for the abstention cost λ in Section 3.3.1. Parameter type

Domain

Default family

Cost ratio / mixing weight Selection fraction k/n Abstention cost (rescaled) Risk-aversion γ (rescaled) Unbounded scale parameter

(0, 1) (0, 1) (0, 1) (0, 1) (0, ∞)

Beta(α, β) Beta(α, β) Beta(α, β) Beta(α, β) LogNormal(µ, σ 2 ) or Gamma(α, β)

For a Beta(α, β) prior with α, β > 1, the mode is (α − 1)/(α + β − 2), and the concentration grows with α + β. The mode can therefore be fixed at a domain-motivated value and the concentration can be controlled independently by scaling α and β together. 33

I.3

Diagnostics

Before reporting PWU results, we recommend the following checks. Visualization Plot the density of πθ and verify that it concentrates mass where expected. We do this in Figure 1. Quantile check Compute the 5th, 50th, and 95th percentiles of πθ . Each should correspond to a plausible deployment scenario. For example, under our πc = Beta(2, 10), the 5th, 50th, and 95th percentiles of c are approximately 0.033, 0.148, and 0.364, corresponding to false-negative-to-falsepositive cost ratios of roughly 29:1, 5.8:1, and 1.7:1. If any of these quantiles seems implausible for the target deployment context, the prior should be reconsidered. Degeneracy check Verify that πθ does not place substantial mass on regions where the decision becomes trivial or where UQ is least needed (e.g., c → 0 or c → 1 for binary decisions, λ → 0 for selective prediction). This is exactly the pathology we identify in Section 3 for the implicit priors of conventional metrics. If the prior is degenerate in this sense, truncate or otherwise reshape it. Sensitivity check Re-run the benchmark with at least one perturbed prior (e.g., shifting the mode by 50%) and verify that conclusions are qualitatively stable. This makes the dependence of results on the prior transparent and guards against overfitting the prior to a particular outcome. Our sensitivity analysis in Appendix J.3 provides a template for such checks. I.4

Worked examples: prior choices used in our experiments

We document our specific prior choices below as worked examples of the procedure in Appendix I.1. All priors were specified a priori from domain heuristics and held fixed across all experiments; they were not selected, tuned, or adjusted based on test-set performance, model rankings, or alignment results. The sensitivity analysis in Appendix J.3 characterizes robustness to misspecification of these choices. Classification For the binary decision problem, our PWU metric coincides with the well-known PSR integral representation (3) discussed in Section 3.2.1. For the form (3), Buja et al. [16] proposed PSRs that use Beta(α, β) priors over c. We follow this approach and set α = 2 and β = 10. These are reasonable choices as the result is a distribution with modal value c = 0.1 and the majority of probability mass on small values of c. This reflects the cost asymmetry in many binary decisions where the cost of a false negative 1 − c is much higher than the cost of a false positive c. In the case of top-k selection, we place a prior over the fraction k/n. For simplicity, we denote this as πk (k), but it should be understood as πk/n (k/n). We again propose a Beta distribution, this time targeting a lower mode of 0.01 (pick top 1%) as many top-k selection studies are about picking only a small number of items from a large dataset. Therefore, we set α = 1.2 and β = 20.8. Regression For selective prediction, we consider the abstention cost to be roughly 10% of the data variance (α = 2, β = 10), so that abstention is often cheaper than the typical squared error but not negligible. For top-k selection, we use the same prior as in classification for k/n, and say that the distribution for γ is parameterized by α = 2 and β = 6.17 This corresponds to a moderate risk-aversion (γ = 0 corresponds to risk-neutrality). We write πc , πλ for Beta(2, 10), πk for Beta(1.2, 20.8), and πϕ for the product density of Beta(1.2, 20.8) and Beta(2, 6) (assuming independence of k/n and γ). On cross-validating the prior We do not cross-validate or otherwise tune the priors on labeled data. PWU priors are not hyperparameters in the conventional sense—they encode a belief about the downstream decision context, which is exogenous to the data. Selecting a prior to maximize alignment on a validation split would conflate “the prior best representing beliefs about a decision scenario ” with “the prior under which a particular model looks best,” reintroducing precisely the misalignment problem PWU metrics are designed to avoid. The appropriate analog of cross-validation 17 Note that although the natural domain of λ and γ is R >0 , we place priors on [0, 1]; in practice, samples are rescaled by

multiplying with the empirical variance of y.

34

is the sensitivity analysis in Appendix J.3, which characterizes robustness to misspecification rather than tuning the prior.

J

Details on our experiments

In both the controlled experiments and the case studies, we aim to measure the alignment between model rankings by a metric M and by a utility Uθ . Throughout all experiments, we fit the same models, use the same metrics M (all metrics from Section 3 and our PWU metrics from Section 4), and apply the same alignment score computation. Model implementations For binary classification, we implement logistic regression (LogReg), random forests (RFs), gradient boosting (GB), sparse variational Gaussian processes (GPs) [47], multilayer perceptrons (MLPs), TabPFN [48], the FT-Transformer [49], SAINT [50], the ResNetMLP [49], and a deep ensemble [51]. For regression, we implement the same suite of models, replacing LogReg with linear regression (LinReg), GB with natural gradient boosting (NGB) [52], and MLPs with heteroscedastic MLPs. Our codebase is primarily based on the following packages: scikit-learn [53] (BSD 3-Clause License), gpytorch [54] (MIT license), ngboost [52] (Apache2.0 license), and tabpfn [48] (Prior Labs License). Linear and Logistic Regression We use standard L2-regularized logistic regression and ordinary least squares, where the predictive variance is analytically available and can be estimated from the data. Random Forests Each ensemble contains 200 trees. Predictive uncertainty combines inter-tree variability (epistemic) and out-of-bag residual error (aleatoric). (Natural) Gradient Boosting We employ gradient boosting for classification and NGBoost [52] for regression, using a Gaussian likelihood. Sparse Variational Gaussian Processes We use the formulation of Hensman et al. [47], employing an RBF kernel with automatic relevance determination. The model is trained by maximizing the variational evidence lower bound (ELBO). Multilayer Perceptrons We train two-layer ReLU networks (128 units per layer) with early stopping. For regression, a heteroscedastic variant outputs both mean and variance, and is trained by maximum likelihood. TabPFN We subsample the training sets to 10,000 instances when necessary, as required by the model. For regression, predictive uncertainty is obtained from the 15.87% and 84.13% quantiles of the predictive distribution, corresponding to ±1 standard deviation under a Gaussian assumption. FT-Transformer We use a [CLS] token, dtoken = 64, 4 attention heads, 3 layers, and dropout 0.1. For regression, the model outputs both a mean µ and a variance σ 2 (via a softplus activation), trained by minimizing the Gaussian negative log-likelihood. SAINT Architecture and training details match those of the FT-Transformer. For regression, a heteroscedastic head is used, analogous to the FT-Transformer. ResNet-MLP We use 3 residual blocks with hidden dimension 128, batch normalization, GELU activations, and dropout 0.1. Each block contains two linear layers with a skip connection. For regression, a heteroscedastic variant with separate mean and variance heads is trained by Gaussian negative log-likelihood. Deep Ensemble Following Lakshminarayanan et al. [51], we train an ensemble of 5 MLPs with distinct random initializations. For classification, the predictive probability is the average of the member probabilities. For regression, each member is a heteroscedastic MLP outputting mean and variance; the ensemble predictive mean and variance are obtained by the standard mixture-ofGaussians formulas. Alignment score computation We now explain how we compute the alignment score for one metric–utility pair (M, Uθ ) on one dataset D, and this follows equally for all considered metric–utility pairs on each dataset. We train and test each model Mj , j = 1, . . . , 5, on D using 5-fold cross-validation (80/20 train–test partition) with fixed random seed, and repeat this train/test loop for 100 different seeds (we have a slightly different setup in the electricity mar35

ket case study, see below). So for each model Mj , we obtain 100 × 5 metric/utility values: {mrep,fold }Mj , {urep,fold }Mj , rep = 1, . . . , 100, fold = 1, . . . , 5. In each of the 100 repeats, we average the metric/utility value over the 5 folds to obtain a stable metric/utility value for each repeat: {m̄rep }Mj , {ūrep }Mj . Using these 100 metric/utility values, we can determine the corresponding model rankings r M,rep = (rM,rep,M1 , . . . , rM,rep,M5 ), r Uθ ,rep = (rUθ ,rep,M1 , . . . , rUθ ,rep,M5 ), rM,rep,Mj , rUθ ,rep,Mj ∈ {0, . . . , 5}. We then measure the alignment of r M,rep , r Uθ ,rep by computing Kendall’s τ . Through this procedure, we obtain 100 metric-utility alignment scores {τ(M,Uθ ),rep }. Choice of Kendall’s τ We aim to evaluate decision-alignment (Definition 3.1) between metrics and downstream utilities. Proposition 3.4 shows that decision-alignment is equivalent to strict orderand tie-preservation between the rankings induced by a metric and a utility. Therefore, evaluating decision-alignment reduces to assessing agreement between the two rankings. Standard measures for ranking similarity include Spearman’s ρ and Kendall’s τ . We choose Kendall’s τ because it is robust and interpretable for small rankings, as in our case. For example, τ(M,Uθ ) = 0.8 means that 80% of all pairwise model comparisons induced by the metric M agree with the downstream utility Uθ . Other possibilities to compare two ranking vectors are top-k agreement or cardinal comparisons (e.g., Pearson’s ρ). As discussed above, these do not evaluate decision-alignment in the sense of Proposition 3.4. That said, these other metrics can be valid for different evaluation goals: top-k if the objective is identifying the best models, and cardinal correlations (e.g., Pearson’s ρ) if magnitudes matter. So as an insightful ablation, we also report the top-1 and top-3 agreement and Pearson’s ρ for our benchmark experiments in Figures J.1, J.2, and J.3. In each case, our PWU metrics perform better overall compared to common UQ metrics.

top-1 agreement

Binary decision

Selective prediction

1.0

1.0

0.5

0.5

0.0

0.0

Top-k selection

Top-k selection

1.0

1.0

0.5

0.5

0.0

0.0

E E L S C et π c πk cc NL B A EC MC -AU E-D M M R not decision-aligned

E E E L C et π λ ϕ NL MS EC MC -AU E-D M M π R

pathological prior

plausible prior (ours)

Figure J.1: Top-1 agreement in classification (left) and regression (right), averaged over our five datasets. The coloring corresponds to our theoretical findings from Section 3. The PWU metrics perform best in identifying the best model for their corresponding utility families.

J.1

Experiments on benchmark datasets

Datasets We use the following benchmark datasets from the UCI repository [55]: bank marketing [56], heart disease [57], ionosphere [58], mushroom [59], and sonar [60] for classification, and air quality [61], auto MPG [62], energy efficiency [63], power plant [64], and wine quality [65] for regression. All datasets use a CC BY 4.0 license. Utilities We measure the alignment of our metrics M to the four utilities Uθ , θ ∈ {c, k, λ, ϕ}, considered in Section 3. We do not settle on one specific value for θ, but sample 5 different θj from the corresponding priors πθ , θ ∈ {c, k, λ, ϕ}, that we define in Appendix I. We then average the 36

top-3 agreement

Binary decision

Selective prediction

1.0

1.0

0.5

0.5

0.0

0.0

Top-k selection

Top-k selection

1.0

1.0

0.5

0.5

0.0

0.0

cc

L BS A

NL

et

E E E L C et π λ ϕ NL MS EC MC -AU E-D M M π R

E E C πc πk EC MC -AU E-D M M R

not decision-aligned

pathological prior

plausible prior (ours)

Figure J.2: Top-3 agreement in classification (left) and regression (right), averaged over our five datasets. The coloring corresponds to our theoretical findings from Section 3. The PWU metrics perform best in identifying the top-3 models for their corresponding utility families.

alignment (Pearson's ρ)

Binary decision

Selective prediction

1

1

0

0

−1

−1

Top-k selection

Top-k selection

1

1

0

0

−1

−1

E E L S C et π c πk cc NL B A EC MC -AU E-D M M R not decision-aligned

E E E L C et π λ ϕ NL MS EC MC -AU E-D M M π R

pathological prior

plausible prior (ours)

Figure J.3: Metric–utility correlation (Pearson’s ρ) in classification (left) and regression (right), averaged over our five datasets. The coloring corresponds to our theoretical findings from Section 3. The PWU metrics exhibit the strongest cardinal metric correlation with respect to their corresponding utility families. alignment scores of a metric M with Uθj , τ(M,Uθj ) across the respective decision family to obtain P5 τ(M,Uθ ) = 15 j=1 τ(M,Uθj ) . We report the median and 5th, 95th percentiles for each dataset in Tables J.1, J.2, J.3, J.4, and visualize the results, averaged over all datasets, in Figure 2. J.2

Applied case studies

Electricity market bidding We consider a wind farm operator who wants to sell electricity in the day-ahead market. In the day-ahead market, bids must be placed one day in advance, at noon, 37

1e6

725000 700000 675000 650000

Frequency

6

Frequency

Cumulative profit

750000

4 2 0

21h LinReg

23h

01h

03h

05h

RF NGB Abstainment

07h

09h

GP Loss

11h MLP

200

100

0 0.0 0.3 0.6 0.9

0.000 0.025 0.050

Threshold

k/n (%)

Kaggle

PAKDD

Lending Club

Figure J.4: Illustrations of our three case studies: cumulative electricity market trading profit over the night of Dec 03 to Dec 04, 2024, showing when which models led to bid abstention or loss (left), Bayes-optimal decision thresholds for the credit customers from our two datasets (center), and the share of accepted loans with a fixed budget B for our 500 different test sets (right). We can see that the PAKDD thresholds and the k/n distribution differ significantly from our chosen priors (see Figure 1)—note that k/n is given in percent. Nevertheless, our PWU metrics perform well (see Figure J.6), demonstrating their robustness under prior misspecification. for each hour h of the next day. Hence, the wind farm operator needs a forecasting model for the next-day electricity generation Eh to decide on the bidding amounts yh , h = 0, . . . , 23. If a forecast is imprecise and the operator bids too much or too little electricity, they risk a penalty for the resulting shortfall or surplus, as, in both cases, the electricity grid’s supply-demand balance needs to be restored. Consequently, when forecast uncertainty is high, it may be better to abstain from bidding, resulting in a typical selective prediction decision problem. In Bruninx et al. [32], the optimization problem for the bidding amount yh∗ for hour h is defined as follows:   max E RhDA yh + RhB (Eh − yh ) yh   s.t. E (Eh − yh )2 ≤ α, 0 ≤ yh ≤ β, where β is the wind farm’s capacity, Eh is the realized electricity generation, RhDA is the day-ahead market price (the unit payoff for the bid quantity yh ), and RhB is the balancing market price at hour h (the unit cost or reward for the deviation Eh − yh ). α is a pre-specified imbalance penalty termed risk certificate. It can range from αmin = Var [Eh ] to αmax = Var [Eh ] + (β − E [Eh ])2 . In their paper, the authors run experiments on several values of the relative α̃ = (α − αmin )/(αmax − αmin ) and we settle on α̃ = 1% for our experiments. Bruninx et al. [32] derive the optimal bidding amount yh∗ for hour h analytically. It is given as      min{E [Eh ] + ∆h ; β} if E RhDA > E RhB , yh∗ = max{E [Eh ] − ∆h ; 0} else , p where ∆h = α − Var [Eh ]. Hence, when the expected balancing price is larger than the expected day-ahead price and the uncertainty (variance) is too high, the optimal decision is abstainment. As in Bruninx et al. [32], we use Belgian balancing price data from [66] and day-ahead price and wind power data (Belwind Phase 1 wind farm) from [67]. The data from [66] is licensed under CC BY 4.0. The data from [67] has no explicit license, but our use complies with their terms and conditions. We train on data from June to November 2024 to predict wind power generation, using the same suite of regression models as in our benchmark experiments. We then employ the trained models to estimate E[Eh ] and Var[Eh ] for each hour h in December 2024, based on which we compute yh∗ and the payoff. To quantify the variability of the metric–utility alignment scores, we bootstrap daily blocks (24 hours) of the 31 days of December, using B = 100 resamples. For each bootstrap replicate, we compute the metrics and utilities on the resampled set of days; the bidding utility is the sum of payoffs over all hours in the replicate, X
Bid-Util = RhDA yh∗ + RhB (Eh − yh∗ ) . h

38

alignment (Kendall's τ)

We show the results in Figure J.5 and Table 2 and see that our PWU metrics exhibit the strongest utility-alignment. On the left of Figure J.4, we show the cumulative payoff on the night of Dec 03 to Dec 04, 2024. Whenever a loss occurred (so payoff < 0), we mark the hour with a cross. Abstainments are marked as circles. At 11 pm, for example, we can see that most models led to a risky trade, resulting in imbalance penalties, while LinReg correctly quantified too high uncertainty and abstained from trading.

Electricity market bidding 1

0

−1

L

NL

E

MS

EC

E

E C et π λ MC R-AU E-D M

conventional metrics

M πϕ

PWUs (ours)

Figure J.5: Metric–utility alignment in the electricity market case study. The PWU metrics are the only metrics with stable positive bidding utility alignment. Credit approval In our second case study, we consider a dataset of credit requests. The decisionmaker (the financial institution) needs to decide for each customer whether to grant or decline a requested credit. The modeling task is probabilistic binary classification, predicting the probability of default (y = 1). We follow the setup from Bahnsen et al. [68], where the cost of a false negative (credit granted that defaults) is 75% of the credit amount, and the cost of a false positive (credit declined that would have been paid back) is the lost profit: CFN,i = Cli · 0.75,

a CFP,i = ri + CFP ,

a where Cli is the customer’s credit line, ri is the loss in profit by rejecting a good customer, and −CFP is the gain through giving out the loan to an alternative customer. Cli and ri are not readily available from the data, but Bahnsen et al. [68] provide a way to calculate these quantities in Appendix A of their paper. The resulting pointwise utility is

−

n X

yi (1 − cFN,i )1fi ≤τi + (1 − yi )cFP,i 1fi >τi ,

i=1 c

where we choose the Bayes-optimal τi = cFP,iFP,i +cFN,i for each customer separately. We show a histogram of the τi in the center of Figure J.4. The problem is similar to a binary decision, but the false negative and false positive costs are customer-dependent and highly variable. We use the same two datasets as in Bahnsen et al. [68]: the PAKDD and Kaggle credit datasets [69, 70]. The Kaggle dataset has no associated license, but our use respects corresponding competition rules. The PAKDD data is openly available through the Python package costcla [71] that uses a BSD 3-Clause License. We show the results on the left of Figure J.6, averaged over the two datasets, and see that the binary decision PWU has the strongest alignment with the credit approval utility. This is particularly remarkable, because we can see in the center of Figure J.4 that the prior πc we chose for our PWU metric is misspecified for the PAKDD dataset. Nevertheless, the PWU metrics robustly achieve strong utility alignment. Peer-to-peer lending In peer-to-peer (P2P) lending, the task is to lend money to a few candidates from a large pool of borrowers, given a fixed budget B. We follow the setup from Byanjankar et al. [72], who use B = 15,000 and the Lending Club dataset [73] that uses a CC0 1.0 Universal license. The modeling task is probabilistic binary classification, predicting the probability of a good loan (y = 1). In our implementation, we greedily select the loans with the highest predicted probabilities until the budget B is exhausted. The utility is then computed as the return through the borrower for those loans that are paid back, minus the lost lent money for those loans that have not been paid back. To make the task nontrivial for our models, we considered only loans with 39

alignment (Kendall's τ)

grade 6 or higher, corresponding to worse gradings. The task resembles top-k selection, but the payoff is customer-dependent and highly variable. In addition, there is no fixed-k, but k is implicitly determined through the fixed budget and the loan amounts of the chosen borrowers—we visualize the empirical k/n-distribution on our 500 different test sets on the right of Figure J.4. We show the results on the right of Figure J.6. We can see that the top-k PWU has the second strongest alignment with the P2P lending utility—even though its prior is misspecified in light of the case study. Credit approval

P2P lending

1

1

0

0

−1

−1

L S cc E E C et π c π k NL B A EC MC -AU E-D M M R conventional metrics

L BS Acc CE CE UC Det M π c M π k E M -A ER PWUs (ours)

NL

Figure J.6: Metric–utility alignment in the credit approval case study (left) and the P2P lending case study (right). We can see that the PWU metrics exhibit the strongest (left) and second strongest (right) alignment with the utilities in these two case studies. Sensitivity analysis

10 5 0

π(γ)

π(k/n)

π(θ)

J.3

0.0 0.2 0.4 0.6 0.8 1.0

0.0 0.2 0.4 0.6 0.8 1.0

θ ∈ {c, λ}

k/n

base

slight

strong

0.0 0.2 0.4 0.6 0.8 1.0

γ

extreme

Figure J.7: Prior perturbations considered in our sensitivity analysis. The base priors anticipated by our PWU metrics are shown in the lightest color, while darker shades correspond to increasing levels of prior misspecification. When constructing a PWU metric for a given decision family, one must commit to a specific prior. In this section, we analyze the sensitivity of PWU metrics to prior misspecification. We rerun our benchmark dataset experiments, but instead of measuring alignment to utilities Uθ originating from the same prior assumption πθ , we measure the alignment to utilities stemming from perturbed priors π̃θ , so that our PWU metric priors are misspecified. We show the choice of our perturbed priors π̃θ in Figure J.7, alongside the base prior our PWU anticipates. We consider a slight, strong, and extreme perturbation. In Figure J.8 and J.9, we show how the metric–utility alignments change across these perturbation levels in the binary classification and regression experiments respectively. Binary decision Our base prior for the binary decision problem has a mode at 10%, reflecting asymmetric costs in which false negatives are substantially more costly than false positives. The perturbed priors shift the mode to 20% (slight), 50% (strong), and 90% (extreme). The metric Mπc is the PWU metric that is decision-aligned with respect to the binary decision problem under the base prior. We can see that a slight perturbation does not affect its utility-alignment, and the strong and extreme perturbation slightly worsens it. However, even after extreme perturbation, our PWU metric is still among the most aligned ones, demonstrating its robustness against prior-misspecification. Notably, even Mπk exhibits fair alignment that is entirely robust against different prior choices for c. 40

Binary decision 1

alignment (Kendall's τ)

0 −1

Top-k selection 1 0 −1

L

NL

c

BS

Ac base

E

E EC

MC

slight

R-A

UC

strong

et

E-D

M πc

M πk

extreme

Figure J.8: Metric–utility alignment in binary classification under increasing levels of prior perturbation. The boxes corresponding to the base prior coincide with those shown in Figure 2. Our PWU metrics exhibit strong robustness even under extreme prior misspecification. Another interesting observation is the development of Acc. Remember that Acc is decision-aligned with respect to the binary decision problem for πc = δ0.5 , so we would expect a strong alignment for the Beta prior with mode at 0.5—which is exactly what we can observe. Top-k selection (classification) Our base prior for the top-k selection problem has a mode at 1%, reflecting the task of selecting a small number of items from a large pool. The perturbed priors shift the mode to 2% (slight), 10% (strong), and 50% (extreme). We first observe that increasing levels of perturbation improve the alignment of metrics that evaluate performance over the entire dataset. This behavior is expected, as larger values of k imply that a greater fraction of instances contributes to the utility Uk . In particular, we expect this trend to be most pronounced for accuracy (Acc), since Acc is decision-aligned w.r.t. Uk when k = n (see Section 3)—which is exactly what is observed. Turning to Mπk , the PWU metric that is decision-aligned with respect to the top-k selection problem under the base prior, we find that slight and strong perturbations even lead to a modest improvement in alignment. A plausible explanation is that for k/n = 1%, the selection task may be trivial on many benchmark datasets, with all evaluated models successfully selecting only positive instances.18 As k increases, the task becomes less trivial, enabling a more meaningful assessment of alignment. Even under extreme perturbation, however, the alignment of Mπk degrades only moderately, demonstrating the robustness of our PWU metric to prior misspecification. Selective prediction Our base prior for selective prediction has a mode at 10% (abstention is cheaper about 10% of the time), while the perturbations consider the modes 20% (slight), 50% (strong), and 90% (extreme). Mπλ is our PWU that is decision-aligned with respect to the selective prediction problem under the base prior. We can see that a slight perturbation does not affect its utility-alignment, and the strong and extreme perturbation slightly worsens it. However, even after extreme perturbation, our PWU metric is still among the most aligned ones, demonstrating its robustness against prior-misspecification. Notably, even Mπϕ exhibits fair alignment that is robust against different prior choices for λ. Another interesting observation is the development of the MSE. Remember that the MSE is decision-aligned for πλ = δ∞ , so we would expect a strong alignment for the Beta prior with a high mode—which is exactly what we can observe. Top-k selection (regression) Our base prior for the top-k selection problem has a mode at k/n = 1% and a moderate risk-aversion parameter γ. The perturbed priors consider k/n = 2% with low risk aversion (slight), k/n = 10% with strong risk aversion (strong), and k/n = 50% with extreme risk 18We nevertheless argue that the base prior remains well motivated, as it reflects realistic application scenarios. Moreover, our case studies demonstrate that even smaller values of k/n can yield non-trivial decision problems depending on the dataset and task.

41

Selective prediction 1

alignment (Kendall's τ)

0 −1

Top-k selection 1 0 −1

L

NL

E

MS

base

E EC

E

MC slight

R-A

UC

strong

et

E-D

M πλ

M πϕ

extreme

Figure J.9: Metric–utility alignment in regression under increasing levels of prior perturbation. The boxes corresponding to the base prior coincide with those shown in Figure 2. Our PWU metrics exhibit strong robustness even under extreme prior misspecification. aversion (extreme). The metric Mπϕ is the PWU metric that is decision-aligned with respect to the top-k selection problem under the base prior. Slight and strong perturbations have essentially no effect on its utility alignment, while extreme perturbation leads to a moderate degradation. Nevertheless, even under extreme perturbation, Mπϕ remains among the most utility-aligned metrics, demonstrating robustness to prior misspecification. Notably, even Mπλ exhibits strong alignment that is robust against different prior choices for ϕ. Overall, these results show that PWU metrics are robust to prior misspecification, further underscoring their suitability in general-purpose UQ benchmarking. Remark J.1 (Utility misspecification robustness). The sensitivity analysis above studies the robustness of a PWU metric Mπ when the prior π is misspecified within its target utility family. A complementary form of misspecification, however, occurs when the entire utility family is misspecified. Our experiments already provide empirical evidence on this scenario, since we evaluate every applicable PWU against every utility family considered. Inspecting Figure 2, we observe that most PWUs remain among the most decision-aligned metrics even when applied across utility families: Mπk exhibits strong alignment with the binary decision utility, and analogously Mπλ and Mπϕ transfer between selective prediction and top-k selection in the regression setting. A plausible explanation is that all of our PWUs are proper scoring rules and thus a fair heuristic in any application considered. Combined with the prior-level sensitivity analysis, this suggests that PWU metrics degrade gracefully under both forms of misspecification, further supporting their use as default metrics in general-purpose UQ benchmarking. J.4

Reproducibility

Our codebase is provided at https://github.com/fortuinlab/prior-weighted-utilities. Essentials on the experiments are described in this manuscript. All further details can be directly extracted from the codebase. The code is readily executable and documented in its README.md. All experiments are deterministic through fixed seeds. We ran all experiments on an internal highperformance compute cluster. For PWU evaluation we used CPU nodes (Intel Xeon Gold 6248R, 2 cores and 60 GB of memory per job), with each evaluation taking less than one minute. For model training we used GPU nodes, where each job was allocated one NVIDIA A100 (40 GB MIG slice), 16 CPU cores, and 60 GB of memory. Training all models in a single repeat–fold combination takes approximately 14–17 minutes for the binary classification, univariate regression, multiclass classification, and multivariate regression benchmarks, and 6–16 minutes for the credit approval and P2P lending case studies. The electricity market case study has no repeat–fold structure and takes 59 minutes for a single full run. Reproducing the full benchmark suite (10 models for binary classification and univariate regression, 5 models for the multiclass and multivariate settings, each 42

across 5 datasets and 500 repeat–fold combinations) and the case studies therefore requires on the order of 2,500 GPU-hours in total. Table J.1: Metric–utility alignment (Kendall’s τ of model rankings) over 100 repeats in the binary decision problem. We mark the highest median in bold. Mπc aligns best with respect to the binary decision utility family on all datasets. NLL

BS

Acc

ECE

MCE

R-AUC

E-Det

Mπc

Mπ k

bank marketing 0.69

0.72

0.62

0.40

0.14

0.74

0.76

0.79

0.58

[0.61, 0.78]

[0.64, 0.81]

[0.46, 0.73]

[0.17, 0.57]

[-0.11, 0.39]

[0.64, 0.80]

[0.63, 0.85]

[0.69, 0.85]

[0.41, 0.72]

heart disease 0.50

0.45

0.24

0.37

0.18

0.34

0.20

0.58

0.09

[0.32, 0.64]

[0.23, 0.62]

[0.02, 0.49]

[0.12, 0.57]

[-0.09, 0.56]

[0.11, 0.52]

[-0.17, 0.40]

[0.41, 0.70]

[-0.16, 0.34]

ionosphere 0.64

0.57

0.42

0.16

-0.03

0.65

0.54

0.73

0.46

[0.45, 0.73]

[0.45, 0.70]

[0.23, 0.59]

[-0.03, 0.34]

[-0.30, 0.29]

[0.56, 0.76]

[0.36, 0.72]

[0.66, 0.84]

[0.28, 0.63]

mushroom 0.65

0.65

0.71

0.61

0.57

0.56

-0.38

0.80

-0.13

[0.54, 0.75]

[0.55, 0.79]

[0.61, 0.94]

[0.52, 0.74]

[0.46, 0.72]

[0.47, 0.72]

[-0.62, -0.24]

[0.66, 0.86]

[-0.43, -0.02]

sonar

K

0.57

0.53

0.40

0.40

0.04

0.44

0.33

0.64

0.21

[0.41, 0.70]

[0.32, 0.65]

[0.18, 0.61]

[0.15, 0.65]

[-0.18, 0.31]

[0.24, 0.60]

[0.03, 0.61]

[0.43, 0.79]

[-0.02, 0.48]

Multiclass and Multivariate Experiments

We now extend our binary and univariate experiments from Section 5.1 to the multiclass and multivariate setting. We consider the metrics and utilities defined in Appendix G. For the experimental evaluation of the multiclass-decision PWU, we use the same Beta(2, 10) prior on c as in the binary case (Appendix I), with j sampled uniformly over [K]. For the experimental evaluation of the multivariate selective-prediction PWU, we use the same Beta(2, 10) prior on the dimensionless abstention-cost factor λ as in the univariate case (Appendix I), with λ rescaled per dimension by the empirical variance inside the utility. As probabilistic ML models, we again use LogReg, RFs, GB, MLPs, and TabPFN for classification and LinReg, RFs, NGBoost, MLPs, and deep ensembles for regression. We use covertype [74], dry bean [75], iris [76], pen-based digits [77], and wine [78] as classification datasets and air quality [61], energy efficiency [63], solar flare [79], SGEMM [80], and parkinsons telemonitoring [81] as regression datasets. All datasets use a CC BY 4.0 license. We can see the results in Figure K.1. In classification, the PWU metric exhibits the strongest utility alignment alongside the BS. In regression, the ES has a stronger median alignment, but a significantly lower, even negative, 5% quantile, while the PWU metric is the only metric with consistent positive utility alignment.

L

Recommendations for UQ benchmarking

Our findings suggest a concrete revision to the standard UQ evaluation protocol. We do not advocate replacing NLL, BS, or ECE—these metrics carry useful auxiliary information about likelihood and calibration—but we do advocate against treating them as the primary evidence that a UQ method is “good.” Specifically, we recommend that papers proposing new UQ methods: 1. Report PWU metrics for at least two distinct decision families that plausibly reflect deployment use cases, e.g., a cost-sensitive binary decision PWU and a selective-prediction PWU. 43

Table J.2: Metric–utility alignment (Kendall’s τ of model rankings) over 100 repeats in the top-k decision problem (binary classification). We mark the highest median in bold. Mπk aligns best with respect to the top-k utility family on most datasets. NLL

BS

Acc

ECE

MCE

R-AUC

E-Det

M πc

Mπ k

bank 0.55

0.63

0.59

0.43

0.16

0.63

0.53

0.57

0.62

[0.40, 0.68]

[0.50, 0.75]

[0.44, 0.72]

[0.16, 0.57]

[-0.08, 0.41]

[0.49, 0.76]

[0.35, 0.70]

[0.44, 0.71]

[0.50, 0.72]

heartdisease -0.17

-0.16

-0.01

-0.16

0.15

-0.02

0.00

-0.15

0.24

[-0.38, 0.13]

[-0.39, 0.10]

[-0.29, 0.34]

[-0.38, 0.08]

[-0.15, 0.46]

[-0.29, 0.32]

[-0.33, 0.38]

[-0.36, 0.12]

[-0.07, 0.49]

ionosphere -0.12

-0.15

-0.33

-0.10

-0.02

-0.16

0.02

-0.16

-0.11

[-0.29, 0.07]

[-0.33, 0.07]

[-0.51, -0.03]

[-0.36, 0.09]

[-0.33, 0.25]

[-0.29, 0.01]

[-0.25, 0.32]

[-0.29, 0.03]

[-0.31, 0.18]

mushroom -0.24

-0.24

-0.02

-0.29

-0.24

0.09

0.69

-0.11

1.00

[-0.42, -0.11]

[-0.42, -0.16]

[-0.42, 0.11]

[-0.42, -0.16]

[-0.42, -0.16]

[-0.29, 0.24]

[0.51, 0.78]

[-0.38, -0.02]

[1.00, 1.00]

sonar -0.07

-0.06

-0.07

-0.14

0.05

-0.04

0.04

-0.08

0.09

[-0.29, 0.12]

[-0.22, 0.14]

[-0.24, 0.11]

[-0.39, 0.11]

[-0.20, 0.26]

[-0.20, 0.14]

[-0.22, 0.25]

[-0.31, 0.19]

[-0.18, 0.31]

alignment (Kendall's τ)

Binary decision

Selective prediction

1

1

0

0

−1

−1

E L BS EC NL not decision-aligned

c M π j,

pathological prior

L NL

E ES M πλ MS plausible prior (ours)

Figure K.1: Metric–utility alignment in classification (left) and regression (right), averaged over our five datasets. The coloring corresponds to our theoretical findings from Appendix G. In classification, the PWU metric exhibits the strongest utility alignment alongside the BS. In regression, the PWU metric is the only metric with consistent positive utility alignment. 2. State the prior π explicitly, following our recommendations in Appendix I: Priors should be elicited from the deployment context where possible and from generic plausible defaults (such as the Beta priors we use). Hidden priors—those implicit in NLL, BS, or accuracy—are exactly what our analysis shows to be problematic. 3. Report conventional metrics alongside, but interpret them in light of their implicit priors (Table 1). NLL improvements at the tails of c may be irrelevant for a deployment in which the cost ratio is plausibly bounded away from 0 and 1; ECE improvements may not translate into utility gains at all. This is a modest amount of additional reporting—a few extra columns in a results table—but it shifts UQ benchmarking from implicitly assuming a single pathological prior to explicitly evaluating against a small panel of plausible decision contexts.

44

Table J.3: Metric–utility alignment (Kendall’s τ of model rankings) over 100 repeats in the selective decision problem. We mark the highest median in bold. Mπλ aligns best with respect to the selective decision utility family on all datasets. NLL

MSE

ECE

MCE

R-AUC

E-Det

Mπλ

Mπ ϕ

air quality 0.39

0.16

0.44

0.47

0.36

0.21

0.67

0.16

[0.30, 0.47]

[0.02, 0.26]

[0.36, 0.51]

[0.39, 0.56]

[0.27, 0.43]

[0.10, 0.31]

[0.63, 0.72]

[0.07, 0.24]

auto mpg 0.20

0.27

0.04

0.08

0.08

-0.42

0.79

0.69

[0.00, 0.31]

[0.11, 0.44]

[-0.12, 0.19]

[-0.06, 0.24]

[-0.04, 0.24]

[-0.63, -0.25]

[0.72, 0.84]

[0.57, 0.78]

energy efficiency 0.81

0.84

0.19

0.22

0.88

-0.43

0.92

0.69

[0.72, 0.88]

[0.79, 0.87]

[0.08, 0.31]

[0.13, 0.33]

[0.82, 0.92]

[-0.56, -0.28]

[0.87, 0.93]

[0.64, 0.75]

power plant -0.27

0.03

-0.02

-0.02

-0.01

-0.03

0.33

0.28

[-0.36, -0.16]

[-0.08, 0.10]

[-0.09, 0.10]

[-0.08, 0.11]

[-0.10, 0.06]

[-0.17, 0.13]

[0.27, 0.42]

[0.22, 0.38]

wine quality -0.48

0.20

-0.08

-0.04

-0.10

0.09

0.66

0.32

[-0.61, -0.35]

[0.02, 0.34]

[-0.20, 0.06]

[-0.16, 0.09]

[-0.25, 0.05]

[-0.05, 0.27]

[0.51, 0.76]

[0.18, 0.44]

Table J.4: Metric–utility alignment (Kendall’s τ of model rankings) over 100 repeats in the top-k decision problem (regression). We mark the highest median in bold. Mπϕ aligns best with respect to the selective decision utility family on all datasets. NLL

MSE

ECE

MCE

R-AUC

E-Det

Mπλ

Mπ ϕ

air quality -0.18

0.24

-0.19

-0.14

-0.19

-0.42

0.24

0.81

[-0.29, -0.09]

[0.14, 0.35]

[-0.26, -0.11]

[-0.23, -0.06]

[-0.32, -0.09]

[-0.50, -0.26]

[0.13, 0.32]

[0.78, 0.85]

auto mpg 0.24

0.28

-0.03

-0.01

0.16

-0.47

0.76

0.96

[0.09, 0.41]

[0.14, 0.48]

[-0.19, 0.19]

[-0.18, 0.22]

[0.01, 0.29]

[-0.68, -0.28]

[0.63, 0.85]

[0.94, 0.99]

energy efficiency 0.72

0.62

0.34

0.40

0.72

-0.21

0.78

0.94

[0.62, 0.82]

[0.55, 0.70]

[0.24, 0.45]

[0.29, 0.50]

[0.66, 0.79]

[-0.32, -0.04]

[0.72, 0.83]

[0.92, 0.96]

power plant 0.31

0.57

0.20

0.13

0.58

0.08

0.87

0.96

[0.16, 0.39]

[0.45, 0.66]

[0.08, 0.29]

[0.00, 0.24]

[0.48, 0.66]

[-0.15, 0.30]

[0.76, 0.92]

[0.93, 0.97]

wine quality -0.01

0.70

-0.53

-0.52

0.32

0.49

0.41

0.87

[-0.20, 0.13]

[0.53, 0.83]

[-0.65, -0.35]

[-0.62, -0.33]

[0.16, 0.44]

[0.32, 0.65]

[0.26, 0.52]

[0.76, 0.94]

45