Dirichlet-Based Monte Carlo Dropout for Uncertainty Estimation in Neural Networks

Dirichlet-Based Monte Carlo Dropout for Uncertainty Estimation in Neural Networks Rouaa Hoblos & Noura Dridi & Noureddine Zerhouni & Zeina Al Masry

arXiv:2605.23635v1 [stat.ML] 22 May 2026

Université Marie et Louis Pasteur, SUPMICROTECH, CNRS, Institut FEMTO-ST, F-25000 Besançon, France [email protected], [email protected], [email protected], [email protected]

Résumé. Les réseaux neuronaux traditionnels fournissent des prédictions déterministes sans estimation explicite de l’incertitude. Bien que les réseaux neuronaux bayésiens (BNNs) offrent une approche théoriquement fondée pour la quantification d’incertitude, leur complexité computationnelle limite leur passage à l’échelle. Le Dropout de Monte Carlo (MC Dropout), initialement introduit comme une technique de régularisation, a été démontré comme une approximation de l’inférence bayésienne en permettant une modélisation probabiliste via plusieurs passes stochastiques. Dans ce travail, nous renforçons l’estimation de l’incertitude en intégrant un cadre basé sur la distribution de Dirichlet au sein de MC Dropout. Plus précisément, nous exploitons la formulation proposée dans Sensoy et al. (2018), où les probabilités de classe sont modélisées par une distribution de Dirichlet, permettant une représentation plus informative de l’incertitude. Notre approche conserve l’efficacité computationnelle du MC Dropout tout en améliorant la qualité des estimations d’incertitude. Nous discutons les fondements théoriques de notre méthode et la comparons aux techniques existantes de quantification d’incertitude. Nos résultats mettent en évidence l’efficacité de la méthode proposée pour produire des estimations d’incertitude bien calibrées, offrant une solution pratique pour les modèles d’apprentissage profond intégrant l’incertitude. Mots-clés. Quantification d’incertitude, Monte Carlo Dropout, Réseau de Neurones, Loi des Grands Nombres Abstract. Traditional neural networks provide deterministic predictions without inherent uncertainty estimates. While Bayesian Neural Networks (BNNs) offer a principled approach to uncertainty quantification, their computational complexity limits scalability. Monte Carlo (MC) Dropout, initially introduced as a regularization technique, has been shown to approximate Bayesian inference by enabling probabilistic modeling through multiple stochastic forward passes. In this work, we enhance uncertainty estimation in deep learning by integrating a Dirichlet-based framework within MC Dropout. Specifically, we leverage the formulation proposed by Sensoy et al. (2018), where class probabilities are modeled using a Dirichlet distribution, allowing for a more informative uncertainty representation. The proposed approach maintains the computational efficiency of MC Dropout while improving the quality of uncertainty estimates. We discuss the theoretical foundations of our method and compare it with existing uncertainty quantification techniques. The results highlight the effectiveness of the proposed method in producing well-calibrated uncertainty estimates, offering a practical solution for uncertainty-aware deep learning models.

JdS2025

1

Keywords. Uncertainty quantification, Monte Carlo dropout, Neural Network, Law of Large Numbers

1

Introduction

Effective decision-making relies on the ability to make precise and reliable predictions including quantifying uncertainty associated with these predictions, this means the confidence level to attribute to these predictions. This is particularly relevant for critical domains such as medical field or autonomous vehicle control. Several approaches have been developed, including Bayesian Neural Networks (BNNs) [1], ensemble methods [2], and single deterministic models [3]. BNNs estimate uncertainty by modeling neural network weights as probability distributions. A variational interpretation of the BNN is proposed in [4]. The main result is that optimizing the loss function of a NN with dropout is equivalent to a Bayesian variational approximation of a Gaussian process. This method enables probabilistic modeling by leveraging dropout at both training and inference stages, allowing multiple stochastic forward passes to estimate uncertainty. The method is computationally efficient compared to classical BNNs [1], which are computationally expensive for posterior inference. On the other hand, ensemble methods train multiple models and quantify uncertainty using the variability among their predictions. Single deterministic methods, such as evidential deep learning [5], use a Dirichlet distribution to represent the class probability. The parameters of this distribution are related to the prediction and uncertainty. In [5], the authors propose a new method called Deep Evidential Classification (DEC), an evidential neural network to learn these parameters in classification tasks. In [6], the author proposes an Information Aware Dirichlet (IAD) network to learn a Dirichlet prior distribution on predictive distributions. In this work, we propose to use the same hypothesis for the class probability, however, we propose a new method to calculate Dirichlet parameters and therefore the uncertainty. This method leverages MC Dropout while maintaining computational efficiency and improving uncertainty estimation through a theoretically grounded probabilistic formulation. The performance of the method is confirmed on tabular and image datasets and compared with the approach proposed in [5] and [6] using different metrics.

2

Proposed approach

Let {(Xi , Yi )}N i=1 be a set of data samples, where Xi represents an input (e.g., an image or a feature vector), and Yi is the corresponding label in a classification task, with Yi ∈ Y = {1, 2, . . . , K}. A neural network parameterized by θ is used to predict the class probability distribution pθ (Xi ) = (p1 , p2 , . . . , pK ), where pk is the predicted probability for class k, obtained by applying the softmax function to the network’s logits. Uncertainty and Evidence. In [5], Sensoy has proposed calculating the uncertainty using evidence learned by the neural network. The NN outputs evidence which is used in the calculation of the parameters of a Dirichlet distribution. These parameters are used to calculate the predicted probabilities and the uncertainties associated to them. The uncertainty is given JdS2025

2

by: u=

K S

(1)

P with S = K k=0 (ek + 1), with ek the evidence derived for the kth singleton. The authors [5] propose to replace the softmax activation function, classically used for classification, by the ReLU function to obtain positive values corresponding to the evidence. In [6], Tsiligkaridis uses the softplus activation function. The predicted probabilities are the expected value calculated using the properties of a Dirichlet distribution: pk = αSk where pk is the probability associated to the k th class. Dirichlet Based Uncertainty Estimation-MC Dropout (DBUE-Dropout). As proposed in [5], the class probability vector p = (p1 , p2 , . . . , pK ) is assumed to follow a Dirichlet distribution parameterized by α = (α1 , α2 , . . . , αK ): K

1 Y αk −1 p p(p|α) = B(α) k=1 k

(2)

where B(α) is the multivariate Beta function. The variance of the Dirichlet distribution can be written as [7]: var(pk ) =

E[pk ](1 − E[pk ]) P . 1 + k αk

(3)

X

E[pk ](1 − E[pk ]) −1 Var(pk )

(4)

Therefore, S would be given by: S=

αk =

k

where E[pk ] is the expected probability for class k and Var(pk ) is the variance associated to this predicted probability. In the context of classification, a NN architecture is designed to output the class probabilities. Besides, by activating the dropout on the test, we obtain a different probability for each new forward. It is worth pointing out that MC dropout computes the standard deviation to estimate uncertainty; however, the standard deviation measures the variation but doesn’t convey the full characteristics of a distribution. We propose to represent the probability of classes by the Dirichlet distribution. Indeed, a classification problem with K labels is a multinomial opinion equivalent to a Dirichlet probability distribution function[8]. Given an input Xi , we perform L stochastic forward passes with dropout enabled, obtaining L probability vectors: p(1) , p(2) , . . . , p(L) . For each class k, the expected probability and variance are computed empirically. Besides, we estimate E[pk ] and Var(pk ) using the Law of Large Numbers (LLN) [9]. The LLN states that the empirical mean of a sequence of independent and identically distributed (i.i.d.) random variables converges to the true expectation as the number of samples increases. Despite its simplicity, this process makes our proposed method a fully coherent Bayesian method with a connection to evidence. Using the empirical estimates, S is computed using (4) and used to obtain the final uncertainty measure as in (1). JdS2025

3

A higher S means more evidence and less uncertainty and vice versa. It is important to note that the uncertainty is calculated using the absolute value of S to ensure it is non-negative. Algorithm. An algorithm of the proposed methodology is shown below (Algorithm 1). Algorithm 1 Dirichlet Based Uncertainty Estimation-MC Dropout Require: Dataset {(Xi , Yi )}ni=1 , trained neural network pθ (X), number of MC Dropout iterations L, number of classes K Ensure: Uncertainty estimates ui for each sample Xi Initialize: Neural network pθ (X) with MC Dropout enabled for each sample Xi in the test set do Collect L stochastic forward passes: (l) pi = pθ (Xi ) for l = 1, . . . , L Determine the predicted  P class:  (l) ci = arg maxk L1 Ll=1 pi (k) Extract probabilities of the predicted class: (l) pi (ci ) for l = 1, . . . , L Compute the empirical PL (l)expectation for predicted class: 1 E[pi (ci )] = L l=1 pi (ci ) Compute the empirical variance: 2 PL  (l) 1 Var(pi (ci )) = L−1 (c ) − E[p (c )] p i i i i l=1 Compute Dirichlet strength parameter for predicted class: i )](1−E[pi (ci )]) Si = E[pi (cVar(p −1 i (ci )) Compute uncertainty: ui = SKi end for Return: Uncertainty estimates {ui }ni=1

3

Results

The proposed method is evaluated on both tabular and image datasets, specifically MNIST and Titanic. To assess the effectiveness of the UQ method in detecting data shifts, we introduce noisy data and Out-of-Distribution (OOD) datasets. For MNIST, Gaussian noise with a mean of 2 and standard deviation of 100 is added to test images, while for Titanic, noise is selectively applied to different features with a mean of either 0 or 1 and a maximum standard deviation of 3. The Fashion-MNIST (FMNIST) dataset serves as an OOD dataset for MNIST, whereas the Forest Fires dataset is used for Titanic. Training is conducted with 350 epochs and 100 Monte Carlo dropout samples (pdrop = 0.25) for Titanic, while MNIST is trained for 150 epochs with 100 dropout samples (pdrop = 0.4). Furthermore the method is compared to DEC [5] and IAD [6]. For the DEC, the NN outputs the evidence calculated using the ReLU activation function, and the uncertainty is calculated using (1). As for the JdS2025

4

IAD, the NN outputs the evidence calculated using the Softplus activation function, and the uncertainty is calculated using (1). Table 1: Performance Metrics Across Titanic Dataset with Different Methods.

Method IAD [6] DEC [5] DBUE-Dropout

Train

Accuracy Test Noisy

OOD

Train

Uncertainty Test Noisy

86.82 88.22 89.6

76.92 72.032 80.42 65.73 77.62 62.94

0.00 0.00 0.19

0.53 0.59 0.027

0.54 0.51 0.012 0.57 0.48 0.0033 0.026 0.028 2.0

OOD

Figure 1: Rotated images with their associated uncertainties. Table 1 illustrates the classification performance as well as the UQ performance using the three methods on Titanic dataset. The uncertainty is calculated per sample and the median over train, test and noisy sets are given (to avoid effect of outliers). It is noted that DEC and IAD are unable to detect OOD data since the uncertainty decreases while the accuracy is decreasing meaning the algorithm is confident in its wrong decisions. Meanwhile, DBUEDropout can efficiently detect the noisy and OOD data. This means accuracy decreases on OOD data and uncertainty increases on noisy and OOD data. As for images, the DBUEDropout method performs well on the MNIST dataset. The method demonstrates high performance with a training accuracy of 99.95%, a test accuracy of 99.25%, a noisy test accuracy of 9.74%, and an OOD accuracy of 6.78%. As for the uncertainty measure, it is 0.27 for training, 0.33 for testing, 0.53 for noisy inputs, and 19.19 for OOD data. The stranger the image becomes to the trained model, the higher the uncertainty. This can be shown by the increase in the median of uncertainty across different datasets. To assess DBUE-Dropout’s performance on images, some were rotated (0–360°), and their uncertainties were analyzed (Figure 1). The digit 0 maintains low uncertainty due to its rotational invariance. As for the second image, uncertainty spikes at ambiguous angles, especially when horizontal, but remains low when clearly resembling a 1 or 7. The third row represents an image of digit 4 highlights that even correct predictions can exhibit high uncertainty. For instance, when inverted, uncertainty remains elevated due to visual ambiguity. However, as JdS2025

5

the digit returns to an upright position, uncertainty drops from almost 14 to 2, reinforcing the model’s confidence in more recognizable orientations.

4

Conclusion

A new method for uncertainty quantification in classification tasks is proposed. The idea is to construct a deep evidential neural network with MC dropout. Given different samples of the class probability, the parameters of the Dirichlet distribution are calculated and used to evaluate the uncertainty. Results confirm the overall performance in terms of classification and uncertainty quantification. Besides, comparison with state-of-the-art methods illustrates the potential of the proposed method to detect distribution shift through noisy and out-ofdistribution (OOD) data across both tabular and image datasets. Future work will focus on enhancing the method to improve performance on noisy data and generalize to imbalanced datasets.

References [1] Radford M. Neal. Bayesian learning for neural networks. 1995. [2] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017. [3] Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. Advances in neural information processing systems, 31, 2018. [4] Gal.Y and Ghahramani.Z. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. 2016. [5] Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classification uncertainty. Advances in neural information processing systems, 31, 2018. [6] Theodoros Tsiligkaridis. Information aware max-norm dirichlet networks for predictive uncertainty estimation. Neural Networks, 135:105–114, 2021. [7] Thomas Minka. Estimating a dirichlet distribution, 2000. [8] Audun Jsang. Subjective Logic: A formalism for reasoning under uncertainty. Springer Publishing Company, Incorporated, 2018. [9] Michel Loève and M Loève. Elementary probability theory. Springer, 1977.

JdS2025

6