C ONFORMALISED IMPRECISE INFERENCE FOR ROBUST
arXiv:2605.25882v1 [cs.LG] 25 May 2026
EXTRAPOLATION UNDER LIMITED DATA
Yu Chen Institute for Risk and Uncertainty University of Liverpool Liverpool [email protected]
Scott Ferson Institute for Risk and Uncertainty University of Liverpool Liverpool [email protected]
A BSTRACT Recent advances in uncertainty quantification increasingly emphasise the distinction between aleatory and epistemic uncertainty in machine learning, motivating the need for more unified frameworks. However, despite much progress in producing reliable predictions, existing methods often lack rigorous guarantees when generalising beyond the training domain. We propose a conformalised imprecise inference framework for robust extrapolation, which is model-agnostic and augments predictive models with imprecision and distance awareness. The proposed approach yields imprecise predictions (probability boxes) that remain valid under distributional shift, maintaining coverage while adaptively expanding uncertainty in extrapolation regimes. Experiments on synthetic and benchmark datasets demonstrate improved robustness and reliable coverage compared to standard probabilistic approaches, particularly under limited data. Keywords Imprecise probability · Probability box · Conformal prediction · Extrapolation · Limited data
1
Introduction
Generalising beyond the data seen from learning is an essential task of machine learning systems [1]. Much progress in uncertainty quantification (UQ) for artificial intelligence (AI) has focused on addressing the overconfidence of trained models in high-stakes settings [2, 3, 4, 1]. Importantly, recent work has increasingly distinguished between aleatory and epistemic sources of predictive uncertainty, with particular emphasis on the latter, focusing on challenges in limited data, measurement noise, model assumptions, distributional shift, and extrapolation [5, 6, 7]. Modern theories of uncertainty, e.g. the imprecise probability framework such as credal sets, evidence theory (Dempster Shafer structures), hierarchical higher-order distribution, uncertain numbers, fuzzy sets etc [8, 9, 10], can represent much weaker statements of knowledge and more diverse types of uncertainty than conventional probability theory to reason under polymorphic uncertainty. Following the rapid development of approximate Bayesian inference in deep learning, these modern theories are applied to comprehensively analyse the uncertainty landscape of the machine learning systems [11, 12, 13, 14]. The pursuit of uncertainty research align with the goal of enhancing the robustness of the AI systems, wherein statistical guarantee for generalisation is desired, even with distribution shift. Epistemic learning theory [15] emphasizes that the system must be designed to be prepared for data it has not yet encountered. Despite much progress towards the guarantee on the generalisation performance, with notable examples including PAC-Bayesian theory [16] and classic conformal prediction [4], it is challenging to produce rigorous bounds that are guaranteed to enclose the ground truth under distributional shift, particularly in extrapolation settings. In this work, we investigate epistemic uncertainty in extrapolation settings, while systematically analysing the impact of limited data through controlled experiments. We propose a conformalised imprecise inference (CII) framework that integrates distance awareness with imprecise probability representations. In particular, we leverage a scalar distance score to characterise deviation from the training data manifold and use it to drive a distance-dependent discrepancy
Conformalised imprecise inference for robust extrapolation under limited data
model. This discrepancy is embedded within a novel ascloseas bounds construction, yielding conformalised p-box predictions that maintain coverage while adaptively expanding uncertainty under distributional shift.
2
Conformalised framework for robust extrapolation
2.1
Background on predictive capability with uncertainty modelling
Consider a general learning problem in a regression setting. Given observation data D = {(xi , yi )N i=1 } ∈ (X × Y) which are assumed to independent and identically distributed (i.i.d) with a distribution PD , the expected loss of a hypothesis h ∈ H : X → Y is given as L = E(x,y)∼PD ℓ(x, y) . To formulate heteroscedastic aleatory uncertainty, neural networks can be viewed as probabilistic models with a data-dependent noise structure p(yi | f ω (xi )) = N µ(xi ), σ 2 (xi ) . The empirical loss, given by the negative log-likelihood, is written as: N
LAU =
1 1 X 1 (y − µ(xi ))2 + log σ(xi )2 N i=1 2σ(xi )2 2
(1)
To model the epistemic uncertainty on model parameters, Bayesian deep learning evaluates the posterior weights p(ω|D) given the observation data, see xx for variants of approximate Bayesian inference to handle the otherwise intractable inference problem. Notably, SVI turns the inference problem into optimisation of a stochastic objective named ELBO by introducing variational distribution qϕ (ω) parameterised by ϕ: LEU = KL[qϕ (ω) ∥ p(ω)] − Eqϕ (ω) log p(D|ω)
(2)
In compounding these two sources, researches differ in a mixture route to get compound distribution [17, 18] or a credal route to obtain imprecise structures [11, 12]. In addition, conformal prediction (CP)[19] presents as a new paradigm for quantifying predictive uncertainty by enriching a prediction from a hypothesis h into prediction sets. 2.2
Conformalised inference with distance awareness and distributional imprecision
Compared to the classic CP where conformity guarantee is constructed on the assumption of exchangeability which may be violated under distributional shift, we proposed an extrapolation-focused approach where in-distribution (ID) and out-of-distribution (OOD) data are explicitly differentiated, in principle allowing the robustness against extrapolation problems on the basis of distributional awareness. Meanwhile, the proposed conformalised framework is also agnostic to any pre-trained model which may possess some capacity of uncertainty quantification. With respect to the trans-probabilistic predictive distributions yielded by the base hyposis, in which the predictive uncertainty could be manifested in different uncertainty models: probability distributions, a set of distributions (e.g. p-boxes), or intervals; whereas the observation may be precise or subject to measurement uncertainty. We employ the stochastic area metric as the conformal score, d(x), suggesting the discrepancy between two random data generating processes, the predictive model p(y|x) and the real-world observation Yobs , under polymorphic uncertainty. That is, in recognising the measurement uncertainty, the observation data could be elicited as intervals (say plus-minus form) or distributions (say PERT distribution). Generally, these are effectively uncertain numbers [10] at varying level of imprecision, and the stochastic area metric [20] in a generalised formulation is given by: d(x) := Ex ∆([F (x), F (x)], [S(x), S(x)])
(3)
where [F (x), F (x)] represent the bounds of the probability box of the prediction, while [S(x), S(x)] represent the empirical probability box from the data. Distance awareness has been identified as a key heuristic to reflect distribution shift [3]. To characterise extrapolation in the covariate space, we project inputs onto a distance space R ∈ R embedding the p deviation of a testing data point from the training data manifold. Specifically, We define the distance score as: r(x) = (x − µ)T Σ−1 (x − µ), which corresponds to the Mahalanobis distance. When Σ = I, this reduces to the Euclidean distance. The score r(x) induces an ordering over inputs that reflect their degree to distributional shift related to the training data. To learn how predictive uncertainty evolves with distance, we construct a calibration subset Dc ⊆ D in a hybrid manner that balances coverage of the input space using distance with an emphasis on boundary regions using the scenario 2
Conformalised imprecise inference for robust extrapolation under limited data
400
600
range
400
[E, E] obs
0.4
−200
−200
ground truth MoG MVE
−400 −5
0
5
(a) MVE and MoG
MVE MoG CII ground truth
0.6
0
0
0.8 Pr(X ≤ x)
200
200
1.0
−400
ground truth CII range
−600
mean band [E, E]
−5
0
0.2 0.0
5
−400
(b) CII bands
−200
0
200
400
600
(c) Range and mean interval
−100
0
100
200
(d) Predictive uncertainty
Figure 1: Demonstration of the predictive capability on extrapolation with a toy example of the cubic function. optimisation [21]. Conformal scores are computed in Dc whereby an interval predictor model (IPM) M : r → Id , is constructed to rigorously characterise the range Id = [d, d] for unseen inputs for generalisation. It benefits from the statistical guarantee of high probability of covering the unseen response [22]. Id (r, P) = [d, d] = {fl (r) ≤ d(r) ≤ fu (r)}
(4)
where M = {d(r) = M (r, q), q ∈ P} encodes a set of models parameterised the set P. Pnu by a vector q ranging Pin nl Practically it is prescribed by its upper and lower boundaries fu (r) = j=0 uj ψj (r) and fl (r) = j=0 lj ψj (r), polynomial functions in which ψj (r) are basis function and nu is the number of coefficients, that fully enclose the response with the minimal spread. This can be solved by a semidefinite program (SDP) with sum-of-squares (SOS) constraints [23] which even allows the distance to be imprecise. We further adopt a distance-aware adjustment rule to reflect a transition from accuracy-oriented in-distribution to conservative uncertainty inflation under extrapolation. Let r0 = supx∈Dtrain r(x) denote the boundary of the training support. We define: dl (r(x)) + du (r(x)) , r(x) ≤ r0 , ˜ = d(x) 2 d (r(x)), r(x) > r0 , u
(5)
Our approach allows the conformal adjustment to depend on the distance score r(x), enabling a distance-aware calibration. As a result, the method retains reliable coverage in the core region while adaptively inflating uncertainty in extrapolation regimes. This provides a principled mechanism to bridge conformal prediction with distribution shift, without requiring explicit modelling of the test distribution. 2.3
Ascloseas bounds with imprecision
We recover the set of all the probability distributions within the area metric d to the prediction of the base hypothesis. It is seeking the credal set of all possible predictions, the ball, under the constraint of given discrepancy measure: Q ∈ Bp (P, Q) = {Q : Wp (P, Q) ≤ d}
(6)
We re-interpret the area metric as the Wasserstein distance and thus exploiting an optimal mass transport plan given the discrepancy budget d. The envelope can be obtained: G(x) =
sup
Q((−∞, x])
(7)
Q((−∞, x])
(8)
Q:W1 (P,Q)≤d,Q∈C
G(x) =
inf
Q:W1 (P,Q)≤d,Q∈C
where C encodes extra information on the admissible distributions in the ball, such as the support, moment, or the shape constraints. This aids in reducing the epistemic uncertainty of Q, serving as additional constraints for the optimisation programs of the Eq. (7) and Eq. (8). 3
Conformalised imprecise inference for robust extrapolation under limited data
2.4
Conformalised imprecise inference
Ideally, a reliable prediction should capture all the possible sources of uncertainty into the final prediction. Probability boxes, turns out to be a useful structure to confound various sources of uncertainty through the mechanism of probability bounding, where its precision (i.e. the breadth γ between the upper and lower edge) suggests the level of epistemic uncertainty. More importantly, the bounds can be accordingly inflated to account for additional aspects of uncertainty or approximation, serving as a principled way to aggregate uncertainties in a unified structure. Based on the ascloseas bounds, our method enriches a base prediction with imprecision and robustness to generalisation, as indicated below: ˜ ∗ )} ∼ [G(x), G(x)] C(x∗ ) = {y : d(x∗ , y) ≤ d(x
(9)
As mentioned before, this framework accepts any form of the base prediction as an uncertain number which may be a distribution, a p-box or an interval, possibly yielded from the models in Section 2.1. To assess the predictive performance of the conformalised imprecise inference (CII), we consider three complementary metrics capturing accuracy, validity, and imprecision. Distance-stratified coverage. We define the empirical coverage rate as: N
1 X I{yi ∈ C(xi )} N i=1
ξ=
(10)
which measures the proportion of observations enclosed by the predictive set. A reliable method should achieve high coverage, particularly under distributional shift. To assess conditional validity with respect to distributional shift, we consider coverage conditioned on the distance score r(x). Since r(x) is continuous, we discretise its range into B bins {Bb }B b=1 . Let Ib = {i ∈ {1, . . . , nval } : r(xi ) ∈ Bb }
(11)
denote the set of validation points whose distance scores fall into bin Bb . We define the distance-stratified coverage as ξb =
1 X I{yi ∈ C(xi )}. |Ib |
(12)
i∈Ib
Quantile-scaled predictive breadth. To of a p-box prediction, we measure the area between R quantify the imprecision its upper and lower bounds, γ(x) = G(t) − G(t) dt, whose magnitude depends on the scale of the response variable, which varies across datasets. We further introduce a scale-adjusted version based on the empirical range of the data. Specifically, let Q0.05 and Q0.95 denote the 5% and 95% quantiles of the response variable. We define the quantile-scaled breadth as γn (x) =
γ(x) . Q0.95 − Q0.05
(13)
This scaling yields a dimensionless measure of predictive imprecision that is robust to outliers and comparable across datasets. Larger values of γn indicate greater epistemic uncertainty in the prediction. Pooled accuracy To obtain an overall performance measure, we aggregate individual p-box predictions into a pooled representation using u-pooling [20], which generalises classical PIT-based calibration diagnostics to imprecise predictions. Each observation is mapped to an interval-valued u via the predicted p-box. Pooling these intervals yields an empirical p-box on [0, 1], and calibration is quantified by the area between this p-box and the uniform distribution. Lower values of dp indicate better calibration. Z 1 dp = 0
GN (u) − GN (u) du. 4
(14)
Conformalised imprecise inference for robust extrapolation under limited data
Boston MoG 0.5
1.0
0.4
0.8
0.3
MVE
CII
OOD
ID 50 40
γn
ξ
dp
0.6
0.2
30 20
0.4 0.1
10
0.2 0.0
0 0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
0.2
0.4
0.6
ε
ε
ε
(a)
(b)
(c)
0.8
Figure 2: Performance comparison on the Boston dataset under varying data proportions ϵ. Panels (a)–(c) report pooled calibration error dp , coverage ξ, and quantile-scaled breadth γq , respectively. Results are shown for in-distribution (ID, dashed) and out-of-distribution (OOD, solid) settings.
3
Experiments
We present empirical evaluations of the proposed method: first on a toy dataset to illustrate extrapolation behaviour and predictive uncertainty, and then on UCI regression benchmarks to assess performance under distributional shift and different data sizes. For CII, calibration points are sampled to span the full range of distance scores while ensuring sufficient representation near the edge of the training support, where extrapolation effects are most pronounced. 3.1
Predictive uncertainty on the extrapolation task
We start with a toy dataset, i.e. the cubic function regression y = x3 + ϵn where ϵn ∼ N (0, 9) [24]. A small sample set of 40 data points are generated from U(−4, 4) but the prediction is required on [−8, 8]. Fig. 1(a) shows the bounds from the established probabilistic predictors. MVE models the heteroscedastic aleatory uncertainty [25], while MoG suggests a mixture of an ensemble of Gaussian distributions obtained through sampling the posterior weights learned from variational inference E = {f ω (x), ω ∼ p(ω|D)}. To maintain the epistemic uncertainty in the set and compound the aleatory and epistemic uncertainty, We take the envelope of the ensemble of distributions as the base hypothesis, the p-box Pbase = [F (x), F (x)], into the CII framework using Eq. (9), in which the lower and upper bounds are respectively F (x) = inf i∈E Fi (x) and F (x) = supi∈E Fi (x). As seen from Fig. (1a), MVE suffers from this small data regime whereas MoG produces good uncertainty estimates for interpolation but fails to rigorously enclose the observation in the extrapolation domain, indicated outside the red vertical bars. In contrast, CII produces high-quality extrapolation, yielding an uncertainty band that not only fully encloses the observations even far beyond the training support (Fig. 1c), but also captures a consistent trend, with the upper boundary continuing to enclose the underlying function (Fig. 1b). Fig. (1d) shows the comparison between the conformalised p-box and the predictive distributions from the other two methods. 3.2
Effects of training data size on uncertainty evaluation
In practice, data sets are often expensive, unreliable or scarce. The availability of data is seen as a major source epistemic uncertainty in casting inference on data-driven models. To comprehensively test and demonstrate the capability of CII approach against limited data and extrapolation, we employed UCI datasets commonly used in previous endeavours [2] of quantifying predictive uncertainty. However, it should be noted that, for the particular objective of evaluating extrapolation performance, we partition the dataset into training and holdout subsets via quantile thresholds of the distance score in a ratio of 3:7. This splitting process is repeated 20 times and the average test performance of each method is reported. In addition, to investigate the impact of dataset size on the learned uncertainty and the resulting predictions, we conduct experiments across varying proportions of the training data, i.e. ϵ = {0.1, 0.3, 0.7, 0.9}, of the training data. Fig. 2 summarises performance on the Boston dataset across varying data proportions. The proposed CII method consistently achieves the lowest pooled discrepancy error dp while maintaining perfect empirical coverage ξ in both 5
Conformalised imprecise inference for robust extrapolation under limited data
Table 1: Cross-dataset performance at fixed ϵ = 0.5 for in-distribution (ID) and out-of-distribution (OOD) evaluation. Dataset
ϵ
Distribution
Boston
0.5
Concrete
0.5
Energy
0.5
Kin8nm
0.5
ID OOD ID OOD ID OOD ID OOD ID OOD ID OOD ID OOD ID OOD
MoG dp
Naval Propulsion 0.5 Power Plant
0.5
Wine
0.5
Yacht
0.5
MVE
ξ
dp
CII
ξ
dp
ξ
γn
0.28 ± 0.00 1.00 ± 0.00 0.24 ± 0.01 0.26 ± 0.02 0.00 ± 0.00 1.00 ± 0.00 1.45 ± 0.03 0.35 ± 0.00 0.94 ± 0.01 0.26 ± 0.01 0.15 ± 0.03 0.00 ± 0.00 1.00 ± 0.00 2.30 ± 0.14 0.16 ± 0.01 1.00 ± 0.00 0.13 ± 0.01 0.70 ± 0.02 0.00 ± 0.00 1.00 ± 0.00 2.49 ± 0.06 0.25 ± 0.01 0.99 ± 0.00 0.17 ± 0.01 0.62 ± 0.03 0.00 ± 0.00 1.00 ± 0.00 13.02 ± 1.58 0.20 ± 0.00 1.00 ± 0.00 0.14 ± 0.01 0.77 ± 0.02 0.00 ± 0.00 1.00 ± 0.00 3.32 ± 0.07 0.19 ± 0.00 1.00 ± 0.00 0.16 ± 0.01 0.62 ± 0.03 0.00 ± 0.00 1.00 ± 0.00 6.41 ± 0.41 0.16 ± 0.00 1.00 ± 0.00 0.11 ± 0.01 0.97 ± 0.01 0.00 ± 0.00 1.00 ± 0.00 4.05 ± 0.05 0.15 ± 0.00 1.00 ± 0.00 0.12 ± 0.01 0.86 ± 0.02 0.00 ± 0.00 1.00 ± 0.00 4.39 ± 0.07 0.24 ± 0.00 1.00 ± 0.00 0.28 ± 0.02 1.00 ± 0.00 0.00 ± 0.00 1.00 ± 0.00 234.64 ± 2.86 0.24 ± 0.00 1.00 ± 0.00 0.35 ± 0.03 0.50 ± 0.06 0.00 ± 0.00 1.00 ± 0.00 241.62 ± 3.22 0.45 ± 0.00 1.00 ± 0.00 0.06 ± 0.00 1.00 ± 0.00 0.00 ± 0.00 1.00 ± 0.00 65.36 ± 0.06 0.46 ± 0.00 1.00 ± 0.00 0.30 ± 0.01 0.73 ± 0.02 0.00 ± 0.00 1.00 ± 0.00 74.95 ± 0.49 0.13 ± 0.00 1.00 ± 0.00 0.16 ± 0.01 0.54 ± 0.02 0.00 ± 0.00 1.00 ± 0.00 3.39 ± 0.09 0.11 ± 0.00 1.00 ± 0.00 0.19 ± 0.01 0.45 ± 0.02 0.00 ± 0.00 1.00 ± 0.00 8.63 ± 0.86 0.13 ± 0.00 0.89 ± 0.01 0.18 ± 0.01 0.69 ± 0.03 0.01 ± 0.00 1.00 ± 0.00 0.69 ± 0.04 0.12 ± 0.00 0.89 ± 0.01 0.18 ± 0.01 0.46 ± 0.02 0.00 ± 0.00 1.00 ± 0.00 1.19 ± 0.11
in-distribution (ID) and out-of-distribution (OOD) settings. This indicates reliable and well-calibrated predictions across data regimes. In contrast, MoG attains high coverage in the ID setting but exhibits degradation under distribution shift, particularly at small data proportions, as reflected by increased dp and reduced OOD coverage. MVE performs substantially worse, with consistently low coverage due to the absence of epistemic uncertainty modelling. These observations are consistent with the cross-dataset results in Table 1, where CII uniformly preserves coverage while adapting predictive imprecision. In particular, the increase in γn under OOD settings (Fig. 2(c)) aligns with the broader trend that CII expands uncertainty under distributional shift, whereas baseline methods fail to maintain validity. Due to limit space, only the metrics associated with ϵ = 0.5 is reported in Table 1. Fig. 2(c) illustrates the increase in predictive imprecision of the conformalised p-box as the available data decreases. With fewer training samples, model uncertainty grows, leading to broader predictive sets. Notably, a clear separation emerges between the ID and OOD regimes, with OOD predictions exhibiting consistently larger imprecision. This distinction is further highlighted in Fig. (3). This awareness of distribution shift is illustrated in Fig. (3), where the empirical CDFs of the quantile-scaled breadth γn exhibit a systematic gap (a rightward shift) from ID (dashed) to the OOD (solid) regime. For a fixed data proportion ϵ, the OOD curves consistently indicate larger predictive breadth, reflecting increased uncertainty under distribution shift. Importantly, this adjustment occurs while maintaining valid coverage, demonstrating that the proposed method encodes distributional mismatch directly into the geometry of the predictive sets. 1.0
Empirical CDF
0.8
0.6
0.4
=0.5 =0.7 =0.9
0.2
OOD ID
0.0 1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
5.0
γn
Figure 3: Empirical CDFs of the quantile-scaled predictive breadth γn for different training proportions ϵ in the Boston dataset. Solid lines correspond to OOD data, while dashed lines correspond to ID data. 6
Conformalised imprecise inference for robust extrapolation under limited data
Importantly, this distance is not used as a hard classifier alone, but serves as a continuous variable that drives a second-order model M(r), which captures the growth of model discrepancy and predictive uncertainty as inputs move away from the training support. By working with this geometry-aware scalar, we obtain a simple yet effective mechanism to characterise extrapolation and modulate uncertainty in a principled manner. Fig. 4 investigates the adaptivity of the proposed conformalised approach based on the distance-stratified coverage, see Eq. (12). That is, it is desired to return larger prediction sets for difficult inputs. While ξb remains approximately constant across distance bins, indicating robust validity, the predictive breadth γn increases with r(x), capturing the growth of epistemic uncertainty under extrapolation. This highlights the effectiveness of distance-aware calibration in maintaining coverage while adapting uncertainty to distributional shift. 1.0
0.6 ξ
200
γn
300
0.8
0.4 100 0.2 Coverage ξ Breadth γn
0.0 3
4 5 Distance score r
0
6
Figure 4: Distance-stratified coverage ξb and quantile-scaled breadth γq versus the distance score r(x) in the wine dataset. In this experiment, no proportion is used to maintain the number of samples at distances. Further to Fig. (4), Fig. (5) displays more intuitively the breadth of the comformalised prediction, which clearly shows the inflation of predictive uncertainty along with the distance. Besides, CII has perfect coverage which delivers rigorous predictions that fully enclose the observation.
1.0
0.6 0.4
Pr(Y ≤ y)
0.8
0.2 0.0 60 30 2.4
0
2.6 Dist
2.8 ance r
−30
3.0 3.2
ion
y
t dic
e
Pr
−60
Figure 5: Difference of imprecision in conformalised predictions along with distance. Observation is indicated as the black dotted line.
7
Conformalised imprecise inference for robust extrapolation under limited data
4
Conclusion and outlook
We proposed a conformalised imprecise inference framework for robust uncertainty quantification under extrapolation. A distance-aware calibration mechanism is used to characterise deviation from the training data manifold and to drive an uncertainty-aware discrepancy model. The method produces prediction sets (probability boxes), realised through the ascloseas bounds, that maintain coverage while adaptively expanding uncertainty under distributional shift. Empirical results demonstrate consistent robustness across datasets and data regimes, particularly in small-data settings. These findings highlight the importance of incorporating distributional awareness into predictive uncertainty. The mechanism of probability bounding provides a natural way to integrate various sources of epistemic uncertainty through inflating the bounds. The existing framework already has some capacity in dealing measurement uncertainty but not rigorously investigated. Future work will explore extensions that more comprehensively address additional epistemic uncertainties including the measurement uncertainty and alternative discrepancy models for improved scalability.
8
Conformalised imprecise inference for robust extrapolation under limited data
A
Stochastic area metric and u-pooling
Stochastic area metric stands for a unified discrepancy measure between two uncertain numbers, which could be embodied as several uncertainty constructs including intervals, probability distributions, probability boxes (p-boxes) and Dempster-Shafer structures (DSS), plus real numbers. Both the model predictions and the observation could be subject to both aleatory and epistemic uncertainty. For each observation yi , the predicted p-box [Fi− , Fi+ ] induces an interval-valued u-score: Ui = [Fi− (yi ), Fi+ (yi )] ⊂ [0, 1].
(15)
This defines a set-valued mapping from observations to the unit interval. Collecting {Ui }N i=1 yields an empirical random set on [0, 1], which can be represented by its lower and upper empirical distribution functions N
GN (u) =
N
1 X 1{Fi+ (yi ) ≤ u}, N i=1
GN (u) =
1 X 1{Fi− (yi ) ≤ u}. N i=1
(16)
These define a p-box [GN , GN ] on [0, 1], which characterises the pooled uncertainty in probability space. Under perfect calibration, this p-box collapses to the uniform distribution. We quantify deviation from calibration by measuring the imprecision of the pooled u-distribution: Z 1 dp = 0
GN (u) − GN (u) du,
(17)
which corresponds to the area between the lower and upper bounds of the aggregated p-box. Smaller values of dp indicate better calibration, with dp = 0 attained when all u-scores are degenerate and uniformly distributed.
References [1] Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine learning, 110(3):457–506, 2021. [2] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050–1059. PMLR, 2016. [3] Jeremiah Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax Weiss, and Balaji Lakshminarayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. Advances in neural information processing systems, 33:7498–7512, 2020. [4] Yaniv Romano, Evan Patterson, and Emmanuel Candes. Conformalized quantile regression. Advances in neural information processing systems, 32, 2019. [5] Eyke Hüllermeier. Learning from imprecise and fuzzy observations: Data disambiguation through generalized loss minimization. International Journal of Approximate Reasoning, 55(7):1519–1534, 2014. [6] Sébastien Destercke. Uncertain data in learning: challenges and opportunities. Conformal and Probabilistic Prediction with Applications, pages 322–332, 2022. [7] Krasymyr Tretiak, Georg Schollmeyer, and Scott Ferson. Neural network model for imprecise regression with interval dependent variables. Neural Networks, 161:550–564, 2023. [8] Arthur P Dempster. A generalization of bayesian inference. Journal of the Royal Statistical Society: Series B (Methodological), 30(2):205–232, 1968. [9] Sébastien Destercke, Didier Dubois, and Eric Chojnacki. Unifying practical uncertainty representations–i: Generalized p-boxes. International Journal of Approximate Reasoning, 49(3):649–663, 2008. [10] Yu Chen and Scott Ferson. Imprecise uncertainty management with uncertain numbers to facilitate trustworthy computations. Python in Science Conference, 2025, 2025. [11] Michele Caprio, Souradeep Dutta, Kuk Jin Jang, Vivian Lin, Radoslav Ivanov, Oleg Sokolsky, and Insup Lee. Credal bayesian deep learning. arXiv preprint arXiv:2302.09656, 2023. [12] Paul Hofman, Yusuf Sale, and Eyke Hüllermeier. Quantifying aleatoric and epistemic uncertainty: A credal approach. In ICML 2024 Workshop on Structured Probabilistic Inference {\&} Generative Modeling, 2024. 9
Conformalised imprecise inference for robust extrapolation under limited data
[13] Alexander Amini, Wilko Schwarting, Ava Soleimany, and Daniela Rus. Deep evidential regression. Advances in neural information processing systems, 33:14927–14937, 2020. [14] Kaizheng Wang, Fabio Cuzzolin, Shireen K Manchingal, Keivan Shariatmadar, David Moens, and Hans Hallez. Credal deep ensembles for uncertainty quantification. Advances in Neural Information Processing Systems, 37:79540–79572, 2024. [15] Shireen Kudukkil Manchingal, Andrew Bradley, Julian FP Kooij, Keivan Shariatmadar, Neil Yorke-Smith, and Fabio Cuzzolin. Epistemic artificial intelligence is essential for machine learning models to truly’know when they do not know’. arXiv preprint arXiv:2505.04950, 2025. [16] Pascal Germain, Francis Bach, Alexandre Lacoste, and Simon Lacoste-Julien. Pac-bayesian theory meets bayesian inference. Advances in Neural Information Processing Systems, 29, 2016. [17] Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems, 30, 2017. [18] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017. [19] Peter Hoff. Bayes-optimal prediction with frequentist coverage control. Bernoulli, 29(2):901–928, 2023. [20] Scott Ferson, William L Oberkampf, and Lev Ginzburg. Model validation and predictive capability for the thermal challenge problem. Computer Methods in Applied Mechanics and Engineering, 197(29-32):2408–2430, 2008. [21] Marco De Angelis, Roberto Rocchetta, Ander Gray, and Scott Ferson. Constructing consonant predictive beliefs from data with scenario theory. In International Symposium on Imprecise Probability: Theories and Applications, pages 357–360. PMLR, 2021. [22] Marco C Campi, Giuseppe Calafiore, and Simone Garatti. Interval predictor models: Identification and reliability. Automatica, 45(2):382–392, 2009. [23] Márcio J Lacerda and Luis G Crespo. Interval predictor models for data with measurement uncertainty. In 2017 American Control Conference (ACC), pages 1487–1492. IEEE, 2017. [24] José Miguel Hernández-Lobato and Ryan Adams. Probabilistic backpropagation for scalable learning of bayesian neural networks. In International conference on machine learning, pages 1861–1869. PMLR, 2015. [25] Laurens Sluijterman, Eric Cator, and Tom Heskes. Optimal training of mean variance estimation neural networks. Neurocomputing, 597:127929, 2024.
10