Uncertainty-Aware Deep Learning for Genomics Applications: Insights from an Empirical Study

Uncertainty-Aware Deep Learning for Genomics Applications: Insights from an Empirical Study Sepideh Saran

sepideh.saran@mdc-berlin.de

The Berlin Institute for Medical Systems Biology, Max Delbrück Center for Molecular Medicine, Germany Department of Electrical Engineering and Computer Science, Technical University of Berlin, Germany

Mahsa Ghanbari

mahsa.ghanbari@mdc-berlin.de

The Berlin Institute for Medical Systems Biology, Max Delbrück Center for Molecular Medicine, Germany Department of Biology, Humboldt University of Berlin, Germany

arXiv:2608.11054v1 [cs.LG] 11 Aug 2026

Uwe Ohler

uwe.ohler@mdc-berlin.de

The Berlin Institute for Medical Systems Biology, Max Delbrück Center for Molecular Medicine, Germany Department of Biology, Humboldt University of Berlin, Germany Department of Computer Science, Humboldt University of Berlin, Germany

Abstract Deep learning models have emerged as the standard computational tool for a wide range of applications in genomics. Yet, uncertainty quantification (UQ) — and more specifically, the reliability of different uncertainty estimates in this domain — has received little systematic attention. This work presents an empirical analysis of UQ in deep learning models, focusing on genomics applications. In a series of experiments, we contrast Deep Ensembles, Bayesian Neural Networks, and Monte Carlo-dropout methods. We assess their ability to quantify uncertainty in different scenarios, accounting for common dataset characteristics in two genomic application areas and modalities: sequence-to-activity models, and single-cell expression analysis. Our systematic comparison framework provides guidelines for the applicability and reliability of UQ methods in genomics, highlighting their strengths and limitations in different scenarios. We show Bayesian Neural Networks are better at capturing uncertainty caused by strong class imbalance and out-of-distribution data in genomics, despite their computational disadvantages. Moreover, we show how uncertainty scores can be used to select high-quality predictions in protein–RNA interactions.

1

Introduction

Neural networks have become the state-of-the-art methods for identifying functional elements in the genome (Eraslan et al., 2019). Superior performance, scalability, and the rise of explainable artificial intelligence (XAI) (Carvalho et al., 2019; Samek et al., 2020; Bach et al., 2015; Sundararajan et al., 2017; Koo & Ploenzke, 2021; Majdandzic et al., 2023) have made deep learning models a popular choice in many genomics applications. For the ultimate use of these models in downstream tasks (e.g. predicting genetic variant effects as part of clinical diagnostics (Monti et al., 2021)), it is essential to provide a measure of confidence in the model’s outputs. Providing uncertainty measurements improve the credibility of a machine learning solution and support downstream decision-making. Moreover, high costs necessitate the transfer of pre-trained models for inference on other datasets, but frequently we lack sufficient information about the characteristics of the original training data (e.g. whether the model was trained on a highly imbalanced/biased dataset). UQ methods can shed light on such shortcomings of the training data. 1

Evaluating UQ in genomics is itself challenging, because of the nature of the data and the diversity of uncertainty sources. Experimental noise, incorrect labels, out-of-distribution samples, and class imbalance are among the major sources of uncertainty for computational models in biology, and they often co-occur with task complexity such as multi-label classification. Critically, reliable ground truth is frequently unavailable — either because labels are themselves derived from noisy experiments, or because the model is applied to a novel domain that lacks expert annotation. This makes assessing the quality of uncertainty scores independently of predictive accuracy essential in genomics. So far, limited work has leveraged UQ methods in genomic applications and provided guidelines for choosing among them. In computational biology, UQ has previously been approached through calibration (Gruber & Buettner, 2022; Maier-Hein et al., 2024), which aligns confidence scores with the true likelihood of predictions, and through conformal prediction (Angelopoulos & Bates, 2021), which provides statistically rigorous coverage guarantees. Both, however, rely on labeled data, making them difficult to apply when labels are noisy or unavailable — both common in genomics. Closest to our work, Bajwa et al. (2024) quantify uncertainty in genomic sequence-to-activity models through the consistency of deep ensemble replicates, and Zhou et al. (2025) introduce DEGU, which distills an ENS into a single model capturing both epistemic and aleatoric uncertainty across functional genomics prediction tasks, with conformal prediction providing coverage guarantees. Systematic benchmarks of UQ methods exist for computer vision and other data-rich domains (Ovadia et al., 2019; Filos et al., 2019; Franzen et al., 2024), but genomics presents fundamentally different challenges that limit the transferability of these findings. Moreover, existing benchmarks typically assess uncertainty through its relation to predictive accuracy in supervised settings, not suitable for the label-scarce settings common in genomics. To the best of our knowledge, no prior work has systematically compared multiple UQ methods across the dataset conditions that characterize genomics, nor provided guidelines for which method to choose in a given setting. To address this gap, we present a framework for evaluating UQ methods in deep learning. We examine UQ through deep ensemble (ENS), Bayesian neural networks (BNN), and Monte Carlo-dropout (MC-dropout), in two well-studied classification tasks in genomics: protein binding prediction from DNA/RNA sequence and predicting cell type from single-cell gene expression. Using both simulated and real data, we systematically compared whether these UQ methods capture uncertainty from various sources commonly encountered in genomics. This controlled comparison highlights the strengths and weaknesses of these UQ methods in different scenarios, providing guidelines for their applicability and reliability. To assess UQ in a controlled manner, we begin with simulated sequence data, where the source of uncertainty can be more reliably controlled. We repeat the experiments with real sequence data and demonstrate the practical utility of uncertainty scores to obtain high-quality predictions. Finally, we look at how the findings play out on a different data modality. The remainder of this paper is organized as follows: In section 2, we introduce the background on the UQ methods and the genomic tasks in the scope of this work. Section 3 describes the framework for this analysis and the modeling details. Section 4, presents highlights of the findings of these empirical evaluations. Finally, section 5 concludes this paper and presents directions for future work.

2

Background

2.1

Methods for quantifying uncertainty in neural networks

To study the predictive uncertainty estimation methods in deep learning, we compare three of the most common UQ methods for neural networks: BNN trained with variational inference, deep ensemble, and MC-dropout. Deep ensembles, inspired by tree-based ensemble models, are a simple way of acquiring uncertainty scores in neural networks (NNs). A deep ensemble is a group of NNs, each trained separately with a different random initialization. The prediction of the ensemble is given by averaging all models’ predictions and the uncertainty score, by taking the standard deviation (SD) or variance of them. Different training strategies have been 2

employed to either train each model in the ensemble on the whole training set or only a subset of it, or to include adversarial training (Lakshminarayanan et al., 2017). Bayesian neural networks are the more theoretically sophisticated alternatives for UQ. BNNs introduce stochasticity to the network by learning a distribution for each weight (and bias) instead of a single point estimate (Blundell et al., 2015). A prior distribution is selected for each parameter and is represented by its mean and standard deviation, increasing the number of trainable parameters in the Bayesian neural network to twice the size of the equivalent deterministic architecture. Since exact inference is computationally intractable for these networks, Variational Inference (Jordan et al., 1999; Wainwright & Jordan, 2008; Graves, 2011; Blei et al., 2017) is one commonly used way to approximate the posterior on weights and thus train the network (Filos et al., 2019). A predictive distribution for the output can also be generated by drawing values for weights from the approximated posterior distribution (i.e., Monte Carlo sampling). Other strategies, such as Laplace approximation and MCMC-based learning, have been used for training BNNs as well (Abdar et al., 2021). Monte Carlo-dropout has been introduced as a probabilistic approach that approximates Bayesian inference in deep Gaussian processes, considering certain assumptions (Gal & Ghahramani, 2016). MC-dropout extends the now-well-established practice of using Dropouts (Srivastava et al., 2014) during the training phase to the test phase, bringing stochasticity to the network at inference time. MC-dropout has the same number of parameters as the deterministic equivalent network and uses the same training procedure, making it computationally more efficient than Bayesian neural networks. MC-dropout has revealed promising results on image datasets (Filos et al., 2019), but to the best of our knowledge, it has not been yet tested on other data types such as genomic sequence data. These three methods represent different trade-offs between theoretical grounding, computational cost, and implementation simplicity. Next, we describe the two genomic use cases in which we compare these methods empirically. 2.2

Use case I: predicting protein binding from DNA/RNA sequence

Proteins regulate different stages of gene expression through binding to specific locations in the genome (DNA) or the transcriptome (RNA). Proteins that bind to the DNA, Transcription Factors (TFs), can activate or repress transcription, and RNA binding proteins (RBPs) can affect post-transcriptional processes such as splicing, RNA localization, translation, and degradation (Gerstberger et al., 2014). These proteins are selective of their target binding sites, as they recognize short, local patterns in the sequence (i.e. sequence motifs) and/or the DNA/RNA structure (i.e. structural motifs). Evolutionarily related proteins often share preferred sequence patterns for binding, while those with different protein binding domains have distinct motifs. Identifying proteins’ binding preferences opens the door to understanding their impact on downstream biological processes. Following previous work Ghanbari & Ohler (2020), we formulate the problem of protein binding prediction as a multi-label classification task (i.e., to predict which protein(s) bind to a given section of DNA/RNA) and extend our experiments to compare the uncertainty estimation approaches for this problem. 2.2.1

Simulated transcription factor dataset

To account for different dataset characteristics of interest in a controlled setting, we generated simulated datasets for predicting protein-binding events for DNA sequences. The input data is a 150-nucleotide long section of DNA, and the classification task is to predict which proteins out of a selection of transcription factors bind to it. We chose 6 TFs with widely known sequence motifs, namely, CTCF, HNF4A, JUN, MEF2A, MYC, and TAL1 as our target classes for this task. With this simulation strategy, we can generate scenarios for comparing UQ method performance on clean versus noisy datasets. In each scenario, the noisy dataset has only a single source of uncertainty introduced compared to the clean version. The simulated input sequences are generated so that the noisy dataset either has missing sequence motifs, or is partially mislabeled. Such cases are common when working with noisy biological data. To this end, we used SimDNA(Shrikumar et al., 2019) to synthesize DNA sequences and plant sequence motifs in them. SimDNA is a tool developed in Python for generating genomic regulatory sequences. It 3

first generates a background sequence and then embeds desired sequence motifs in the background. For our experiments, we used a first-order Markov model to generate DNA background sequences of length 150 and embedded a single motif sequence for a TF class. We used the Jaspar open-source database(Fornes et al., 2020) to obtain sequence motifs of our selected TFs. JASPAR provides manually curated TF binding profiles as position frequency matrices (PFMs). These PFMs are then used in SimDNA to sample and embed a sequence motif of a TF into the background DNA sequence (see suppl. Figure 1). We generated three simulated datasets: a clean single-label balanced dataset as a noise-free reference, a noisy dataset where the sequences have no motifs, and a single-label balanced dataset where a portion of data is mislabeled. The simulated dataset consists of 2000 samples per class equally distributed to training, validation, and test sets (the entire dataset size is 12000). For the dataset with no motifs, we simply used the background sequences without implanting any motifs and for the mislabeled dataset, we swapped labels between the two TF classes of JUN and MEF2A for 30% of class samples. 2.2.2

Real RNA binding protein dataset

In further experiments, we assess the UQ methods on previously published real protein binding datasets. PAR-CLIP experiments provide high-resolution binding site sequences for the RBP of interest (Hafner et al., 2010). These bindings sites are small RNA fragments of typically a few dozen nucleotides, which contain sequence patterns of typically 4-8 nucleotides in length. We select PAR-CLIP data (Mukherjee et al., 2019) for three RBPs, namely MBNL1, PUM2, and QKI, for this task. Since PAR-CLIP datasets are generally noisy, we have selected this subset of RBPs as their preferred sequence motifs are known, making it a suitable choice for benchmarking with real data in a more controlled setting. Our dataset is a combination of several published PAR-CLIP datasets obtained in the HEK293 cell-line, which were processed uniformly through a single pipeline (Corcoran et al., 2011). The dataset contains approximately 16, 500 samples in total. It is imbalanced and includes approximately 475 multi-label samples, meaning one sequence can have the binding motifs for multiple RBPs. For further analysis, we sub-sample from the larger classes to build a balanced multi-label dataset containing about 4, 500 (including 460 multi-label) samples (see Figure 1) and a balanced single-label version of the same dataset. Another source of uncertainty we experiment with in this use case is out-of-distribution data. Novel organisms are exactly the setting where a UQ method matters most in practice: experimental labels are expensive or unavailable, as for a newly sequenced virus, and because the underlying biology is not yet fully understood, one cannot know in advance how much a model trained on human data will transfer, even between datasets from the same protocol and tissue. We tested this directly by applying the RBP models, without retraining, to eight coronavirus genomes where no RBP binding label exists (Section S2.7). 2.2.3

Protein binding prediction

The base model for this use case is a simple deterministic convolutional neural network (CNN) adopted from DeepRipe (Ghanbari & Ohler, 2020), which has state-of-the-art performance for this classification task. The base model architecture has two convolutional layers, each followed by a max-pooling layer and a final fully-connected layer. For single-label setting, we use the softmax activation function in the last layer, whereas, in multi-label, sigmoid. 2.3

Use case II: predicting cell type from single-cell gene expression

Single-cell RNA-seq (scRNA-seq) experiments measure the degree to which each gene is expressed (i.e. activated) for each of thousands of single cells. A typical scRNA-seq experiment measures the expression of around 15, 000 genes for tens of thousands of cells. These data can be used as input for computational tasks such as cell type prediction, understanding disease states, developmental trajectories, and more. Out of technical reasons, data generated by such experiments provides a noisy, large, sparse count matrix (number of cells by number of genes) with many missing values. To analyze this data and derive meaningful biology, it is necessary to present the data in a lower-dimensional space. Dimensionality reduction approaches for scRNA-seq data include classical methods such as variations of PCA, UMAP (McInnes et al., 2018), and matrix factorization methods. More recently, the field has adapted variational auto-encoders 4

Figure 1: Simulated TF (a) and Real RBP (b) datasets: class distribution of the imbalanced version of the simulated TF dataset (a), and of the real RBP dataset (b) that are used throughout the experiments. (VAEs) (Kingma & Welling, 2013) as a powerful alternative, capable of learning biologically meaningful low-dimensional representations in an unsupervised manner (Luecken & Theis, 2019). The expectation is that successful embeddings represent the cell types in latent space while preserving biologically meaningful variation(Rautenstrauch & Ohler, 2024). 2.3.1

Single-cell dataset

The data for this use case is selected from a dataset of multi-modal single-cell experiments (Luecken et al., 2021), available from the Gene Expression Omnibus under accession GSE194122. This dataset was originally published for the multimodal single-cell data integration competition in NeurIPS 2021 (Lance et al., 2022). It contains several single-cell data modalities from a total of 120, 000 bone marrow cells measured on 4 different sites (i.e. laboratories) from 10 human donors. This dataset also provides the annotations for the cell types that are the labels in our use case. To analyze the uncertainty scores in a controlled setting without unwanted source of uncertainty, we select a subset of this dataset to simplify the classification task. Specifically, we selected a subset of four cell types from the single-cell RNA-seq data, but included data obtained from across multiple sites and donors to keep the biological variation. We generate a balanced (180, 000 data points) and an imbalanced version (24, 298 data points) of this dataset (see Figure 3) to compare the performance of UQ methods when imbalance is introduced to a dataset as a source of uncertainty. 2.3.2

Cell type classification

As mentioned above, cell type prediction typically starts from dimensionality-reduced data rather than the raw high-dimensional sparse data. We used a published VAE model Liam(Rautenstrauch & Ohler, 2024) for generating 20-dimensional embeddings from the high-dimensional original gene expression matrix. This model showed state-of-the-art performance in the competition that provided the data for our task(Lance et al., 2022) (Figure 2). We then use the embeddings as input to a two-layer fully-connected neural network model to predict cell types. For the UQ comparison, we generate the embeddings once, and we introduce stochasticity only to the CNN. Therefore, the UQ comparisons here refer to the fully-connected neural network architecture, with 10 nodes in each dense layer and a drop-out rate of 0.3.

3

The empirical analysis framework

The empirical framework outlined in this study intends to provide guidelines for choosing the most suitable UQ method in applications with specific dataset characteristics, such that the calculated uncertainty score can be trusted when using the model in a critical task. To this end, we conduct experiments in each use case under a collection of controlled dataset settings to cover scenarios with a particularly interesting source of uncertainty 5

Figure 2: Single-cell gene-expression dataset: Low-dimensional embeddings. UMAP visualization 20-dimensional embeddings computed by Liam (Rautenstrauch & Ohler, 2024). Left: the full scRNA-seq dataset. Right: the four selected cell-type classes that make up the use case.

Figure 3: Single-cell gene-expression dataset: Class distribution for the four selected cell types in the imbalanced setting.

6

in biological applications, namely highly imbalanced data, scarce data, label noise, out-of-distribution, and multi-label classification. 3.1

Modeling and measuring uncertainty

The base model for solving each task is a simple deterministic neural network as described in Sections 2.2.3 and 2.3.2. We implemented probabilistic versions of these architectures to build uncertainty-aware models (i.e. BNN, MCD and ENS) for each task, respectively. We evaluate the prediction and uncertainty scores of the three selected approaches with respect to the deterministic variant for each scenario. In training the deterministic baseline models, we use dropout for regularization. The BNN is built on a similar architecture, but with Gaussian prior and posterior on the trainable parameters and mean-field variational inference (Peterson & Anderson, 1987; Blundell et al., 2015) as the training approach. To speed up the training process, we use the Flipout Monte Carlo estimator that de-correlates the gradients within each mini-batch (Wen et al., 2018). The MC-dropout networks have the same architecture as the deterministic models, with the difference of activating the dropout in the final layers at testing time. The ENS is a collection of 10 deterministic models, each following the base model architecture. Training plans are adapted per model and dataset characteristic, as each setting requires specific considerations to converge reliably. Training is done to the point where all models reach a similar accuracy on the same setting, for uncertainty scores and predictions to be comparable. All models except for BNN are trained with a batch size of 128 for 50 epochs with early stopping based on the validation loss with the patience set to 5. However, for the results of the networks to be comparable, we needed to train the BNNs longer to reach similar accuracy to MC-dropout’s. For all experiments, we used the Adam optimizer (Kingma & Ba, 2015). After training the probabilistic networks, the inference on the test set is repeated for 100 iterations, building an output distribution for each test example through Monte Carlo sampling. The mean of the distribution is regarded as the model prediction, and the standard deviation represents the model’s uncertainty for the given test sample (see Figure 4). We fix the test set across all models for each task and compare the performance and uncertainties of the models. The experiments are repeated with 5 different random seeds, and the models are implemented in python using Tensorflow (Abadi et al., 2015). 3.2

Evaluating performance and uncertainty scores

We consider the accuracy of the deterministic model as our baseline performance in each scenario. The probabilistic models are then trained to reach similar performance, to ensure comparability, and to avoid sacrificing accuracy when estimating uncertainty. In addition to the standard metrics for evaluating predictive accuracy on a test set, inspired by related work (Lakshminarayanan et al., 2017), we used Negative Log Likelihood (NLL) and the Brier score (Brier, 1950) as two appropriate scoring rules to evaluate the predictive uncertainty. Furthermore, we calculate the Pearson correlation and Spearman correlation between the means of the predictive distributions to illustrate the agreements of the methods in their predictions. The use cases are solved in supervised settings; therefore, we examine the distribution of uncertainty scores both for all test samples and, separately, for the samples classified correctly by both methods of a pair. We assess a potential shift in the distribution of the uncertainty scores’ distribution when we introduce a single source of uncertainty (e.g. mislabeled samples) to clean data. This controlled approach enables us to judge how well the UQ methods can reflect each source of uncertainty that could be present in the data. The range of absolute values for the uncertainty scores generated by the UQ methods can differ for the same data and task across the methods. Moreover, the uncertainty score can be calculated differently on the distribution of the predictions for the same UQ method. Therefore, we are not relying on the absolute values of the uncertainty scores for our evaluation, but we argue that looking at the shift of this range can provide model-and-use-case-specific insights. This exercise helps in choosing a threshold in the downstream use of uncertainty scores to filter for reliable predictions. 7

Figure 4: Overview of the UQ assessment framework. We compare three methods for uncertainty quantification in neural networks, namely, deep ensemble, Bayesian neural network, and the Monte Carlo-dropout. For each task and dataset setting, the three probabilistic networks are trained to achieve similar performance on the same training set. For each trained model, inference on the same test set is repeated 100 times to generate a predictive distribution for each test sample. The mean of this distribution is the prediction and the standard deviation is the uncertainty score for the sample.

8

We note that evaluating the uncertainty score beyond its relation to the prediction accuracy is limited to empirical comparisons on carefully constructed use-cases: there is no ground truth value for the uncertainty score, even for simple supervised tasks. To investigate the agreement of uncertainty estimates provided by the UQ methods and identify commonalities and differences, we compare the uncertainty scores across methods in three ways: Firstly, we calculate Kendall’s tau (Kendall, 1938) between SDs of the pairs of models over 5 random seeds. Kendall rank coefficient correlation (Kendall’s tau) measures the correspondence between ordinal data. We used this test to measure agreement of the ranking of uncertainty scores among UQ models. A strong agreement between ranked list of uncertainty scores in each scenario and random seed indicates that UQ methods point to the same samples as highly uncertain. Secondly, we perform the Kolmogorov–Smirnov (KS) test on the SD distributions of each method separately, comparing a reference (i.e. clean or balanced) dataset setting against a perturbed one, to measure how strongly each method’s uncertainty responds to a given source of uncertainty. The null hypothesis of the test is that, for a given method and class, the predictive standard deviations are drawn from the same distribution in the reference and the perturbed setting. A larger KS statistic therefore means the method’s uncertainty scores for that class moved further when the perturbation was introduced; a value near zero means they did not respond to it at all. Since the KS test can only show if the distribution has moved or not, we complement that with plotting the SD distribution shifts to check if the change in distribution is in the expected direction (i.e. larger uncertainty scores for models train on perturbed data). Lastly, we select the most uncertain samples (i.e highest standard deviation) as identified by each method on the same data and task. We then calculate the overlap of the top 100 most uncertain samples across the experiments as a proxy to check if the UQ methods reflect similar approximations of the uncertainty and point to the same underlying source of uncertainty.

4

Results

To assess uncertainty quantification in a controlled manner, we first study sequence-based prediction of protein binding, where simulated transcription factor data lets us control the source of uncertainty exactly. In parallel, we examine how the same behavior plays out on real RNA-binding-protein (RBP) data. We then move to a different data modality, single-cell gene expression (GEX), evaluating the same aspects and referring back to the sequence-based results. Throughout, we report (i) predictive performance, (ii) agreement of uncertainty scores across UQ methods, and (iii) sensitivity of the uncertainty scores to a single, controlled source of uncertainty.

4.1

Predictive performance

As intended, all models reach high and comparable predictive performance in each task, so that differences in their uncertainty scores can be attributed to the UQ method rather than to differences in accuracy (Figure 5b). On the simulated TF data all three methods likewise reach comparable accuracy, 0.93–0.95 on the clean balanced dataset and 0.79–0.82 once 30% of the JUN and MEF2A class labels are swapped (Figure 5a), confirming that the label noise is what degrades performance rather than any difference between the UQ methods. BNNs require substantially longer training to reach comparable performance. Especially in the multi-label setting, BNN tends towards prediction values closer to zero or one than the deterministic and MCD models, and does not reach comparable performance to the other models in three of the five random seeds on balanced multi-label data, where its accuracy ranges from 0.42 to 0.90 (median 0.69) while MCD and ENS are stable across seeds and vary by less than 0.02. As for the single-cell gene expression task, all UQ methods reach high and comparable accuracy on cell-type prediction, again ensuring that the uncertainty comparison is not confounded by differences in performance (Figure 5c). BNN has again lower performance on the imbalanced version of the same dataset in two of five seeds, reaching 0.74 and 0.81 against 0.98 for MCD; the remaining three seeds are closely comparable. Average precision, which unlike accuracy does not depend on a decision threshold, gives the same ordering of methods and settings (Figure 5d–f).

9

Simulated TF (DNA)

Real RBP (RNA)

Accuracy

a) Accuracy

Single-cell GEX

b) Accuracy

0.9

c) Accuracy

1.0

1.0

0.8

0.9

0.6

0.8

0.8

Average precision

0.4 MCD BNN ENS d) Average precision 1.0

0.9

0.8 MCD g) NLL

BNN

MCD BNN ENS e) Average precision

MCD BNN ENS f) Average precision

1.0

1.000

0.9

0.995

0.8

0.990

0.7

0.985

ENS

MCD h) NLL

BNN

ENS

MCD i) NLL

BNN

ENS

MCD BNN l) Brier score

ENS

0.6

NLL

0.4 0.3

0.2

0.2

0.2 MCD BNN j) Brier score

Brier score

0.4

0.4

ENS

MCD BNN k) Brier score

ENS 0.3

1.0

0.2

0.2 0.5

0.1

0.1 0.0 MCD

BNN

ENS

0.0 MCD

BNN

ENS

MCD

BNN

ENS

balanced, clean

balanced, single-label

balanced, single-label

balanced, mislabelled

imbalanced, single-label

imbalanced, single-label

balanced, multi-label imbalanced, multi-label

Figure 5: Predictive performance of the three UQ methods across all three use cases. Accuracy (a–c), average precision (d–f), negative log-likelihood (g–i), and Brier score (j–l) on the held-out test set for MCD, BNN and ENS. Columns group the use cases: simulated TF binding (a, d, g, j), real RBP binding (b, e, h, k), and single-cell cell-type prediction (c, f, i, l). Box plots summarize 5 random-seed data splits and model initializations.

10

4.2

Agreement of uncertainty scores

We next ask whether the three UQ methods agree on which samples are uncertain. We measure agreement in three complementary ways introduced in Section 3: the rank correlation of uncertainty scores (Kendall’s tau), the overlap of the top-100 most uncertain samples per class, and the KS test on the SD distributions. Across the simulated TF data (Figure 7a,b) and the single-label real RBP settings (Figure 7e,f), Kendall’s tau is positive in all scenarios, with p-values rejecting the null hypothesis of no association. Agreement is consistently stronger between BNN and ENS than between either method and MCD, and it decreases as the task becomes more complex, e.g. in the multi-label setting: in the two multi-label RBP settings (Figure 7g,h) and on balanced single-label single-cell data (Figure 7c), MCD shows no association with either BNN or ENS (median τ between −0.07 and −0.03, significant in only 2–4 of 5 seeds), while BNN and ENS remain positively correlated throughout. Values slightly below zero should be read as an absence of association rather than as inverse agreement. The overlap of the top-100 most uncertain samples on the real RBP data (Figure 6e–h) shows the same pattern: BNN and ENS identify the most similar set of uncertain samples in each setting, and the overlap shrinks with task complexity. The same ordering holds on the simulated TF data (Figure 6a,b), where BNN and ENS share 73–82 of their top-100 most uncertain sequences per TF against 25–77 for the pairings involving MCD. This is expected, as the methods rest on different posterior approximations and therefore need not agree on the exact ranking of the SDs. The KS test (Figure 8c,d) shows that every method’s uncertainty distribution shifts when class imbalance is introduced, but by very different amounts, which we return to in Section 4.3. Together these results show that the choice of UQ method matters, and that agreement between methods can itself be informative when no ground-truth labels are available. The agreement results on the GEX data follow the sequence data in the imbalanced setting but not in the balanced one. The overlap of the top-100 most uncertain samples (Figure 6c,d), Kendall’s tau (Figure 7c,d) and the KS test (Figure 8b) all show BNN and ENS agreeing more closely with each other than with MCD, as on the RBP data. On balanced single-label data, however, MCD’s uncertainty ranking carries no shared signal with either method (median τ = −0.03), whereas the corresponding balanced single-label RBP setting gives τ = +0.41. Agreement then increases rather than decreases once imbalance is introduced, the median τ rising to +0.34 and +0.36 for the two MCD pairings and the mean top-100 overlap from 12 to 26 samples. Both observations trace back to the same cause, which we examine next. MC-dropout’s predictive standard deviation is identically zero for a large fraction of test samples, from 2% on balanced single-label single-cell data to 68% in the worst setting, so for those samples it provides no ranking at all. Its tau values are therefore not comparable across settings, and the gap between the BNN–ENS pairing and the two MCD pairings is wider than the raw values suggest (Section S2.6). A second consequence is that for the minority Erythroblast class the BNN and ENS top-100 sets are disjoint in three of five seeds (Figure 6d). This is a property of the standard deviation as a score rather than a disagreement about which cells are uncertain, and it affects three of 96 class-by-method combinations.

11

100

80

80

80

80

60

60

60

60

40

40

40

40

20

20

20

20

0

0

0

0 iv at ed hr ob la st

M

ac t

14 + CD

CD

4+

yt

Er

yt

K N

iv ac t

T

4+

CD

CD

14 +

M

CT H CF N F4 A J M UN EF 2A M YC TA L1

on o

100

at ed hr ob la st

100

on o

100

K

d) imbal. single

N

c) bal. single

T

b) mislabelled

CT H CF N F4 A J M UN EF 2A M YC TA L1

common in top 100

a) clean

Single-cell gene expression

Er

Simulated TF binding (DNA)

Real RBP binding (RNA)

common in top 100

e) bal. single

f) imbal. single

g) bal. multi

h) imbal. multi

100

100

100

100

80

80

80

80

60

60

60

60

40

40

40

40

20

20

20

20

0

0 1 BNL

M

2 PUM

QKI

0 1 BNL

M

2 PUM

MCD-BNN

QKI

NL1 MB

MCD-ENS

0 PU

M2

QKI

NL1 MB

2

PUM

QKI

BNN-ENS

Figure 6: Overlap of the most uncertain samples identified by each pair of UQ methods. For each class, the 100 test samples with the highest uncertainty score were selected independently per method; bars give the number common to both members of a pair. Panels cover the simulated TF binding task (a, b), the single-cell task (c, d) and the real RBP task (e–h). Bars are the mean over 5 random seeds, error bars one standard deviation.

12

Simulated TF binding (DNA)

Kendall's tau

1.00

a) clean

1.00

Single-cell gene expression

b) mislabelled

1.00

c) bal. single

1.00

0.75

0.75

0.75

0.75

0.50

0.50

0.50

0.50

0.25

0.25

0.25

0.25

0.00

0.00

0.00

0.00

MCD BNN

MCD ENS

BNN ENS

MCD BNN

MCD ENS

BNN ENS

MCD BNN

MCD ENS

BNN ENS

d) imbal. single

MCD BNN

MCD ENS

BNN ENS

Real RBP binding (RNA)

Kendall's tau

1.00

e) bal. single

1.00

f) imbal. single

1.00

g) bal. multi

1.00

0.75

0.75

0.75

0.75

0.50

0.50

0.50

0.50

0.25

0.25

0.25

0.25

0.00

0.00

0.00

0.00

MCD BNN

MCD ENS

BNN ENS

MCD BNN

MCD ENS

BNN ENS

all samples

MCD BNN

MCD ENS

BNN ENS

h) imbal. multi

MCD BNN

MCD ENS

BNN ENS

correctly classified

Figure 7: Agreement of the uncertainty scores of each pair of UQ methods, measured by Kendall’s tau. Rank correlation between the standard deviations assigned to the same test samples, for the simulated TF binding task (a, b), the single-cell task (c, d) and the real RBP task (e–h). Within each panel, values are shown for all test samples and for the subset classified correctly by both methods. Box plots summarize 5 random seeds; the gray line marks τ = 0.

13

Simulated TF binding (DNA)

Single-cell gene expression b) class imbalance 1.0

0.8

0.8

KS statistic

KS statistic

a) partial mislabelling 1.0

0.6 0.4 0.2 0.0

0.6 0.4 0.2 0.0

MCD

BNN

ENS

MCD

BNN

ENS

Real RBP binding (RNA) d) imbalance, multi-label 1.0

0.8

0.8

KS statistic

KS statistic

c) imbalance, single-label 1.0

0.6 0.4 0.2 0.0

0.6 0.4 0.2 0.0

MCD

BNN

ENS

MCD

distribution shifted (p < 0.05)

BNN

ENS

not shifted (p ≥ 0.05)

Figure 8: Two-sided Kolmogorov–Smirnov test on the uncertainty-score distributions. For each method, the predictive standard deviations for a given class are compared between a reference setting and a perturbed one: clean balanced versus 30% mislabeled balanced for TF (a), balanced versus imbalanced for the single-cell data (b), and balanced single-label versus imbalanced single-label for RBP (c), and balanced multi-label versus imbalanced multi-label for RBP usecase(d). Each point is one class in one of 5 seeds; filled points reject the null at p < 0.05. A larger KS statistic therefore means the method’s uncertainty scores for that class moved further when the perturbation was introduced; a value near zero means they did not respond to it at all.

14

4.3

Sensitivity to sources of uncertainty

We now ask whether a UQ method reflects a specific source of uncertainty more strongly than the others. We take a clean single-label balanced dataset as the reference, introduce one source of uncertainty at a time, and ask whether the distribution of uncertainty scores shifts relative to the clean case. The simulated TF data is ideal for this because we control exactly which source is present. Here, we introduced label noise (swapping JUN and MEF2A labels for 30% of samples) and missing motifs (background sequences with no planted motif). Figure 9 shows the resulting shifts in the SD distribution, on a log2 scale. Raw-scale figures are available in supplementary Figure S8, and Figure S9. The missing-motif experiment is reported in Section S2.3. BNN produces the clearest and most directed increase in uncertainty for the two mislabeled classes: its median log2 (SD) rises by 6.8 units on JUN and MEF2A but only 2.7 on the four classes whose labels were untouched. ENS shows the same pattern more weakly (4.8 against 3.2), and MCD is essentially undirected (4.1 against 3.7).For the missing-motif case the methods behave similarly, every class shifting by a large and roughly equal amount, which is expected when the predictive signal is removed from the input altogether. We did not find a consistent winner for mislabeled or missing-motif noise in terms of the size of the shift; the difference between the methods lies in whether the shift is specific to the affected classes. On the real RBP data, however, BNN reflects class imbalance in the most useful direction: when the model is trained on a highly imbalanced dataset, BNN’s uncertainty shifts towards higher values in all three classes, while the deep ensemble becomes more confident (Figure 10). We quantified the shift over all methods, classes and 5 random seeds with the KS test (Figure 8c,d), which rejected the null hypothesis of no shift in 38 of 45 single-label and 28 of 45 multi-label tests, confirming that the uncertainty distribution does move when imbalance is introduced. Across all use cases and perturbations, 176 of 239 tests reject the null. The KS statistic measures magnitude but not direction, and here direction is what matters: a UQ method is only useful if imbalance makes it more uncertain. Comparing the median log2 (SD) between settings, per class and per seed, BNN is the only method that moves towards higher uncertainty in both use cases. ENS moves the other way, becoming more confident when trained on imbalanced data (negative in every class and seed on RBP), which is why its KS statistic is the largest of the three despite reflecting imbalance least usefully. MCD is erratic, its sign flipping between seeds, and it becomes markedly more confident on Erythroblast (−4.6) — the one class the imbalance removed. What matters is not only that the distribution moves, but that it moves most for the classes the imbalance harmed, and the two use cases test this differently. The single-cell setting depletes its minority class, cutting Erythroblast from 2673 to 587 training cells (−78%); the RBP setting leaves its smallest class untouched (QKI, 816 to 808) and creates the imbalance by enriching the majorities instead (MBNL1, +786%). Only the single-cell experiment therefore asks whether the depleted class becomes the most uncertain one. It does, and only for BNN: the Spearman correlation between the change in a class’s training-set size and the shift in its uncertainty is −0.80, negative in all five seeds. For MCD (+0.20) and ENS (+0.40) the sign is reversed, so the classes that gained data are the ones whose uncertainty rises most (Section S2.5). BNN is the only method whose uncertainty tracks which classes the training set under-represents. The sensitivity result also transfers across modalities. As on the RBP data, all three methods respond to class imbalance in the cell type prediction case, but only BNN does so in the direction that makes the score useful (Figure 10). Among the two smallest classes, Erythroblast and CD4+ T activated, BNN shows a clear directed shift towards higher uncertainty, and the KS test confirms that both BNN and ENS respond far more strongly than MCD, whose distribution for CD4+ T activated is statistically indistinguishable between the two settings in four of five seeds (median KS 0.048, against 0.529 for BNN and 0.942 for ENS). Measured by KS magnitude alone ENS responds most in all four classes (Figure 8b), the same pattern as on the single-label RBP data — a further illustration that magnitude and direction have to be read together. BNN is therefore the method that better reflects imbalance and it does so consistently across very different data types. The same pattern extends to a genuinely out-of-distribution case. BNN again shows the clearest response when the model trained on humane genome is tested on coronavirus genome: against a 95th-percentile human threshold it flags 36% of viral windows as unusually uncertain, versus 27% for ENS and only 7% for MCD (Figure 11). Section S2.7 of the supplement elaborates on the out-of-distribution experiments. 15

MCD

BNN

ENS

CTCF

HNF4A

JUN

MEF2A

MYC

TAL1 −80

−60

−40

−20

−40

log2(SD)

−20

0

log2(SD) balanced, clean

−40

−20

0

log2(SD)

balanced, mislabelled

Figure 9: Sensitivity of uncertainty scores to label noise on simulated TF data. Distribution of uncertainty scores when the model is trained and tested on clean balanced single-label data (green) versus the same model under partially mislabeled training data (orange; label swap between JUN and MEF2A). Rows are the six transcription factors; columns are MCD, BNN, and ENS. Scores are on a log2 scale. Results are shown for random seed 7. The cross-modality agreement shown in Figure 10 and finding demonstrated in Figure 11 are central empirical finding of this study: BNN is the only method that consistently reflects both class imbalance and distribution shift, across two data types and two distinct sources of uncertainty.

16

a)

MCD

BNN

ENS

MBNL1

PUM2

QKI

−80

−60

−40

−20

−30

log2(SD)

−10

0

−40

log2(SD)

balanced, single-label

b)

−20

−20

0

log2(SD)

imbalanced, single-label

MCD

BNN

ENS

CD14+ Mono

CD4+ T activated

Erythroblast

NK

−60

−40

−20

−20

log2(SD)

−15

−10

−5

log2(SD)

balanced, single-label

−8

−6

−4

−2

log2(SD)

imbalanced, single-label

Figure 10: Sensitivity of uncertainty scores to class imbalance for the real RBP binding and single-cell gene expression. Columns are MCD, BNN, and ENS; green is the model trained on balanced training set, orange is the model trained on the imbalanced one. All models tested on balanced test set. a) real RBP binding use case, rows are MBNL1, PUM2 and QKI; b) single-cell gene expression use case, rows are CD14+ Mono, CD4+ T activated, Erythroblast and NK. Scores are on a log2 scale, for random seed 11.

17

BNN

ENS

SARS-CoV-1

MCD

0.5

1.0

1.5

0

200

400

0

200

400

0.0

0.5

1.0

1.5

0

200

400

0

200

400

0.0

0.5

1.0

1.5

0

200

400

0

200

400

0.0

0.5

1.0

1.5

0

200

400

0

200

MERS-CoV

SARS-CoV-2*

SARS-CoV-2

0.0

SD (×10

−6

)

SD (×10 human test set

−3

)

SD (×10

400 −3

)

virus genome

Figure 11: Uncertainty increases on out-of-distribution sequences (real RBP use case). Models trained on human single-label RBP data and tested on coronavirus genomes as an out-of-distribution case. Green: uncertainty distribution on the human (in-distribution) test set; orange: on the out-of-distribution data. Rows are virus genomes and columns are UQ methods; SARS-CoV-2* denotes the second SARS-CoV-2 assembly (wuhCor1). Note the different multipliers on the horizontal axes: MC-dropout’s scores are three orders of magnitude smaller than those of BNN and ENS, so the columns must not be compared on absolute values. Results are shown for random seed 7.

18

4.4

Filtering predictions by uncertainty

The controlled experiments above use small datasets for a fair and direct comparison. To test the practical impact of UQ scores on a larger, noisier task, we selected 27 RBPs of varying class sizes from PAR-CLIP experiments and trained a BNN to predict RBP binding, since we showed certain advantages of BNN over MCD and ENS in Section 4.3. Figure 12a) shows the class sizes and imbalance for this dataset. Model architecture, hyper-parameters, and pre-processing are chosen as in the benchmark experiments above. We define a high-quality prediction as one with an uncertainty score below 0.1 for the predicted class (note that we expect this threshold to be application- and dataset-dependent). Filtering on this criterion improves average precision per class (Figure 12b)): it retains 65% of the predictions and raises AP in 20 of the 27 classes, with the mean over classes rising from 0.20 to 0.22, and the precision-recall curves for 5 individual RBPs show the corresponding gain (Figure S3). This demonstrates that uncertainty scores can be put to direct use in selecting reliable predictions for downstream analysis. The uncertainty score also tracks how much data each protein contributed. Grouping the scores by predicted class (Figure 12c), the median uncertainty rises steadily as the training-set class size falls: it is zero for MBNL1, the largest class with 7354 training sequences, and 0.22 for PAPD5, the smallest with 148. Across all 27 classes, the Spearman correlation between training-set size and median uncertainty is −0.59 (p = 0.001), and the five smallest classes have a median 57 times that of the five largest. This is the class-imbalance behaviour discussed in Section 4.3. Section S2.2.2 of the supplement examines what drives the uncertainty on this task: incorrectly classified sequences carry no localised motif evidence, and the model’s errors concentrate between RBPs that belong to the same protein complex, where the single-label ground truth is itself partly ambiguous.

5

Conclusion and Future Work

This work presents a structured framework for evaluating UQ methods in deep learning, one of the first of its kind for genomics applications. It provides initial guidelines for selecting the most appropriate among several popular methods. We systematically compared these methods across two genomics modalities DNA/RNA sequence and (single-cell) gene expression - focusing on dataset characteristics particularly relevant to genomics, such as strong class imbalance, out-of-distribution data, and label noise. We introduce two complementary evaluation perspectives that do not rely on ground truth labels. First, we assess the agreement between UQ methods by identifying the most uncertain samples and measuring their overlap, an approach useful when working with noisy labels or applying pre-trained models to new unlabeled datasets. Second, we evaluate UQ quality through distributional shifts in uncertainty scores when a single source of uncertainty is introduced to otherwise clean data. To the best of our knowledge, no prior study has applied either of these perspectives to compare UQ methods in genomics. From our observations, BNN, if converged, better reflects sources of uncertainty commonly encountered in biological applications, particularly class imbalance and out-of-distribution data. BNN and ENS show higher agreement with each other than either does with MCD. The gap is larger than it first appears: MC-dropout’s uncertainty score is identically zero for a substantial share of test samples in most settings, so in several scenarios it carries no shared signal with the other two methods at all rather than merely less. The practical cost of the three methods differs. BNNs need longer training and more computational resources in every scenario, and they were sensitive to the optimizer and the learning rate, with some runs in the multi-label setting not converging until we lowered the learning rate and allowed longer patience for early stopping. MC-dropout was insensitive to these choices, so the better sensitivity we observe for BNN comes with a real cost in tuning effort (Section S1.1.3). We further demonstrate that uncertainty scores can be used to filter predictions, improving average precision in a realistic 27-class RBP binding task. These findings provide practical guidelines for computational biologists in choosing UQ methods based on their data characteristics, and ways to go about evaluating uncertainty estimates beyond it’s relation to model accuracy. Our empirical framework comes with limitations, which we set out here. Our comparison covers two genomics modalities and a fixed set of experimental scenarios, so which method performs best should be read as a property of these settings rather than as a general ranking. The present study evaluated UQ in a classification context, whereas many applications in genomics involve regression or unsupervised settings. Our study is also limited to neural network-based models, yet many actively used computational genomics solutions are 19

a) 1000

test-set class size

800 600 400 200 PAPD5

EIF3G

EIF3A

EIF3D

RTCB

G35

G45

QKI

UPF1

ZFP36

CPSF4

FXR1

CPSF3

LIN28A

P53_NONO

LIN28B

DICER1

XPO5

PUM2

TAF15

FBL

AGO3

SSB

NOP58

NOP56

FUS

MBNL1

0

b) all predictions uncertainty score < 0.1

average precision

0.8 0.6 0.4 0.2

G45

G35

RTCB

G45

G35

RTCB

PAPD5

QKI QKI

PAPD5

UPF1 UPF1

EIF3G

ZFP36 ZFP36

EIF3G

CPSF4 CPSF4

EIF3A

FXR1 FXR1

EIF3D

CPSF3 CPSF3

EIF3A

LIN28A LIN28A

EIF3D

P53_NONO P53_NONO

LIN28B

TAF15 TAF15

LIN28B

FBL FBL

DICER1

AGO3 AGO3

DICER1

SSB SSB

XPO5

NOP58 NOP58

PUM2

NOP56 NOP56

XPO5

FUS FUS

PUM2

MBNL1 MBNL1

0.0

uncertainty score (predicted class)

c) 10−2 10−4 10−6

Figure 12: Uncertainty filtering on the 27-class real RBP task. a) Number of test samples per RBP class, ordered by class size. b) Average precision (AP) per class for a BNN trained and tested on this dataset, for all predictions and for the subset whose uncertainty score is below 0.1 for the predicted class. c) Uncertainty score of every test sequence, grouped by the predicted class. The axis is logarithmic: the score spans seven orders of magnitude, and MBNL1’s median is zero.

20

still based on more traditional ML or statistical methods. On the other hand, future work should clearly evaluate how UQ methods play out on foundation models, e.g. when using the embeddings as input for smaller models for task-specific fine-tuning, expanding on the single-cell gene expression scenario covered here. We focused on aggregated predictive uncertainty, not differentiating between epistemic and aleatoric uncertainty(Hüllermeier & Waegeman, 2021; Kendall & Gal, 2017), which remains an important direction for future work. Finally, UQ methods produce different absolute ranges of uncertainty scores, and thresholds for filtering predictions will likely need to be determined in an application- and dataset-specific manner. Taken together, our empirical analysis provides a novel framework for evaluating uncertainty estimation methods; we expect that its insights will contribute to an increased adoption of UQ within the practical context of genomics applications. Acknowledgments S. Saran and U. Ohler were supported by the Initiative and Networking Fund of the Helmholtz Association (Helmholtz AI project UNITY, ZT-I-PF-5-149) as well as by the Helmholtz-Einstein International Berlin School in Data Science (HEIBRiDS). M. Ghanbari and U. Ohler were supported by DFG research unit DeSBI (KI-FOR 5363).

References Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, et al. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL https://www.tensorflow.org/. Software available from tensorflow.org. Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U. Rajendra Acharya, Vladimir Makarenkov, and Saeid Nahavandi. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information Fusion, 76:243–297, 2021. ISSN 1566-2535. doi: https://doi.org/10.1016/j.inffus.2021.05.008. URL https://www.sciencedirect.com/science/article/pii/S1566253521001081. Anastasios N. Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distributionfree uncertainty quantification. arXiv preprint arXiv:2107.07511, 2021. Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS ONE, 10(7), 2015. doi: 10.1371/journal.pone.0130140. Ayesha Bajwa, Ruchir Rastogi, Pooja Kathail, Richard W. Shuai, and Nilah M. Ioannidis. Characterizing uncertainty in predictions of genomic sequence-to-activity models. In Proceedings of the 19th Machine Learning in Computational Biology (MLCB) meeting, volume 261 of Proceedings of Machine Learning Research, pp. 279–297. PMLR, 2024. David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association, 112(518):859–877, Apr 2017. ISSN 1537-274X. doi: 10.1080/01621459.2017.1285773. URL http://dx.doi.org/10.1080/01621459.2017.1285773. Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks, 2015. Glenn W. Brier. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1): 1–3, 1950. doi: 10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2. Diogo V. Carvalho, Eduardo M. Pereira, and Jaime S. Cardoso. Machine learning interpretability: A survey on methods and metrics. Electronics, 8(8), 2019. ISSN 2079-9292. doi: 10.3390/electronics8080832. URL https://www.mdpi.com/2079-9292/8/8/832. 21

David L Corcoran, Stoyan Georgiev, Neelanjan Mukherjee, Eva Gottwein, Rebecca L Skalsky, Jack D Keene, and Uwe Ohler. Paralyzer: definition of rna binding sites from par-clip short-read sequence data. Genome Biology, 12, 2011. doi: 10.1186/gb-2011-12-8-r79. Gökcen Eraslan, Žiga Avsec, Julien Gagneur, and Fabian J. Theis. Deep learning: new computational modelling techniques for genomics. Nature Reviews Genetics, 20:389–403, 2019. doi: 10.1038/s41576-019-0122-6. Angelos Filos, Sebastian Farquhar, Aidan N. Gomez, Tim G. J. Rudner, Zachary Kenton, Lewis Smith, Milad Alizadeh, Arnoud de Kroon, and Yarin Gal. A systematic comparison of bayesian deep learning robustness in diabetic retinopathy tasks, 2019. O. Fornes, J. A. Castro-Mondragon, A. Khan, R. van der Lee, X. Zhang, P. A. Richmond, B. P. Modi, S. Correard, M. Gheorghe, D. Baranašić, W. Santana-Garcia, G. Tan, J. Chèneby, B. Ballester, F. Parcy, A. Sandelin, B. Lenhard, W. W. Wasserman, and A. Mathelier. Jaspar 2020: update of the open-access database of transcription factor binding profiles - seventh expansions, 2020. Jannik Franzen, Claudia Winklmayr, Vanessa E. Guarino, Christoph Karg, Xiaoyan Yu, Nora Koreuber, Jan P. Albrecht, Philip Bischoff, and Dagmar Kainmueller. Arctique: An artificial histopathological dataset unifying realism and controllability for uncertainty quantification. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2024. URL https://arxiv.org/abs/2411.07097. Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning, 2016. Stefanie Gerstberger, Markus Hafner, and Thomas Tuschl. A census of human rna-binding proteins. Nature Reviews Genetics, 15:829–845, 2014. doi: 10.1038/nrg3813. Mahsa Ghanbari and Uwe Ohler. Deep neural networks for interpreting rna-binding protein target preferences. Genome Research, 30:214–226, 2020. doi: 10.1101/gr.247494.118. URL https://genome.cshlp.org/ content/30/2/214. Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359–378, 2007. doi: 10.1198/016214506000001437. Alex Graves. Practical variational inference for neural networks. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 24, pp. 2348–2356. Curran Associates, Inc., 2011. URL https://proceedings.neurips.cc/paper/2011/ file/7eb3c8be3d411e8ebfab08eba5f49632-Paper.pdf. Sebastian G. Gruber and Florian Buettner. Better uncertainty calibration via proper scores for classification and beyond. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, 2022. Markus Hafner, Markus Landthaler, Lukas Burger, Mohsen Khorshid, Jean Hausser, Philipp Berninger, Andrea Rothballer, Manuel Ascano, Anna-Carina Jungkamp, Mathias Munschauer, Alexander Ulrich, Greg S. Wardle, Scott Dewell, Mihaela Zavolan, and Thomas Tuschl. Transcriptome-wide identification of rna-binding protein and microrna target sites by par-clip. Cell, 141(1):129–141, 2010. ISSN 0092-8674. doi: https://doi.org/10.1016/j.cell.2010.03.009. URL https://www.sciencedirect.com/science/article/ pii/S009286741000245X. J.L. Jr. Hodges. The significance probability of the smirnov two-sample test. Arkiv för Matematik, 3(43), 1958. Jessica I. Hoell, Erik Larsson, Simon Runge, Jeffrey D. Nusbaum, Sujitha Duggimpudi, Thalia A. Farazi, Markus Hafner, Arndt Borkhardt, Chris Sander, and Thomas Tuschl. RNA targets of wild-type and mutant FET family proteins. Nature Structural & Molecular Biology, 18(12):1428–1431, 2011. doi: 10.1038/nsmb.2163. 22

Marc Horlacher, Svitlana Oleshko, Yue Hu, Mahsa Ghanbari, Giulia Cantini, Patrick Schinke, Ndeye Marame Vercruysse, Shayne D. Wierbowski, Haiyuan Yu, Elisa Bortoletto, Debora Rosati, Melissa Blaschke, Annalisa Marsico, and Uwe Ohler. A computational map of the human-SARS-CoV-2 protein–RNA interactome predicted at single-nucleotide resolution. NAR Genomics and Bioinformatics, 5(1):lqad010, 2023. doi: 10.1093/nargab/lqad010. Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods. Machine Learning, 110(3):457–506, 2021. doi: 10.1007/s10994-021-05946-3. Michael Jordan, Zoubin Ghahramani, Tommi Jaakkola, and Lawrence Saul. An introduction to variational methods for graphical models. Machine Learning, 37:183–233, 01 1999. doi: 10.1023/A:1007665907178. Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? In Advances in Neural Information Processing Systems (NeurIPS), volume 30, 2017. M. G. Kendall. A new measure of rank correlation. Biometrika, 30(1/2):81–93, 1938. ISSN 00063444. URL http://www.jstor.org/stable/2332226. Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015. Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes. arXiv preprint arXiv:1312.6114, 2013. Peter K. Koo and Matt Ploenzke. Improving representations of genomic sequence motifs in convolutional networks with exponential activations. Nature Machine Intelligence, 3(3):258–266, 2021. doi: 10.1038/ s42256-020-00291-x. Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30, pp. 6402–6413. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/ paper/2017/file/9ef2ed4b7fd2c810847ffa5fa85bce38-Paper.pdf. Christopher Lance, Malte D. Luecken, Daniel B. Burkhardt, Robrecht Cannoodt, Pia Rautenstrauch, Anna Laddach, Aidyn Ubingazhibov, Zhi-Jie Cao, Kaiwen Deng, Sumeer Khan, Qiao Liu, Nikolay Russkikh, Gleb Ryazantsev, Uwe Ohler, NeurIPS 2021 Multimodal data integration competition participants, Angela Oliveira Pisco, Jonathan Bloom, Smita Krishnaswamy, and Fabian J. Theis. Multimodal single cell data integration challenge: Results and lessons learned. In Douwe Kiela, Marco Ciccone, and Barbara Caputo (eds.), Proceedings of the NeurIPS 2021 Competitions and Demonstrations Track, volume 176 of Proceedings of Machine Learning Research, pp. 162–176. PMLR, 06–14 Dec 2022. URL https: //proceedings.mlr.press/v176/lance22a.html. Yann LeCun, León Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. In Proceedings of the IEEE, volume 86, pp. 2278–2324, 1998. URL http://yann.lecun.com/ exdb/publis/pdf/lecun-98.pdf. Amy S. Y. Lee, Philip J. Kranzusch, and Jamie H. D. Cate. eIF3 targets cell-proliferation messenger RNAs for translational activation or repression. Nature, 522(7554):111–114, 2015. doi: 10.1038/nature14267. Malte D. Luecken and Fabian J. Theis. Current best practices in single-cell RNA-seq analysis: a tutorial. Molecular Systems Biology, 15(6):e8746, 2019. doi: 10.15252/msb.20188746. Malte D Luecken, Daniel Bernard Burkhardt, Robrecht Cannoodt, Christopher Lance, Aditi Agrawal, Hananeh Aliee, Ann T Chen, Louise Deconinck, Angela M Detweiler, Alejandro A Granados, Shelly Huynh, Laura Isacco, Yang Joon Kim, Dominik Klein, BONY DE KUMAR, Sunil Kuppasani, Heiko Lickert, Aaron McGeever, Honey Mekonen, Joaquin Caceres Melgarejo, Maurizio Morri, Michaela Müller, Norma Neff, Sheryl Paul, Bastian Rieck, Kaylie Schneider, Scott Steelman, Michael Sterr, Daniel J. Treacy, Alexander Tong, Alexandra-Chloe Villani, Guilin Wang, Jia Yan, Ce Zhang, Angela Oliveira Pisco, Smita 23

Krishnaswamy, Fabian J Theis, and Jonathan M. Bloom. A sandbox for prediction and integration of DNA, RNA, and proteins in single cells. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL https://openreview.net/forum?id=gN35BGa1Rt. Lena Maier-Hein, Annika Reinke, Patrick Godau, Minu D. Tizabi, Florian Buettner, Evangelia Christodoulou, Ben Glocker, Fabian Isensee, et al. Metrics reloaded: recommendations for image analysis validation. Nature Methods, 21(2):195–212, 2024. doi: 10.1038/s41592-023-02151-z. Antonio Majdandzic, Chandana Rajesh, and Peter K. Koo. Correcting gradient-based interpretations of deep neural networks for genomics. Genome Biology, 24(1):109, 2023. doi: 10.1186/s13059-023-02956-3. Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426, 2018. Remo Monti, Pia Rautenstrauch, Mahsa Ghanbari, Alva Rani James, Uwe Ohler, Stefan Konigorski, and Christoph Lippert. Identifying interpretable gene-biomarker associations with functionally informed kernel-based tests in 190,000 exomes. bioRxiv, 2021. doi: 10.1101/2021.05.27.444972. URL https: //www.biorxiv.org/content/early/2021/05/27/2021.05.27.444972. Neelanjan Mukherjee, Hans-Hermann Wessels, Svetlana Lebedeva, Marcin Sajek, Mahsa Ghanbari, Aitor Garzia, Alina Munteanu, Dilmurat Yusuf, Thalia Farazi, Jessica I Hoell, Kemal M Akat, Altuna Akalin, Thomas Tuschl, and Uwe Ohler. Deciphering human ribonucleoprotein regulatory networks. Nucleic Acids Research, 47:570–581, 2019. doi: 10.1093/nar/gky1185. neptune.ai. neptune.ai: experiment tracker, 2024. URL https://neptune.ai. Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua V. Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems (NeurIPS), 2019. Carsten Peterson and James R. Anderson. A mean field theory learning algorithm for neural networks. Complex Systems, 1:995–1019, 1987. J. Quinonero Candela, C. E. Rasmussen, F. Sinz, O. Bousquet, and B. Schölkopf. Evaluating predictive uncertainty challenge. In J. Quiñonero Candela, I. Dagan, B. Magnini, and F. d’Alché Buc (eds.), Machine Learning Challenges: Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Textual Entailment, pp. 1–27, Berlin, Germany, April 2006. Springer. doi: 10.1007/11736790_1. Pia Rautenstrauch and Uwe Ohler. Liam tackles complex multimodal single-cell data integration challenges. Nucleic Acids Research, 52(12):e52–e52, 06 2024. ISSN 0305-1048. doi: 10.1093/nar/gkae409. URL https://doi.org/10.1093/nar/gkae409. Wojciech Samek, Grégoire Montavon, Sebastian Lapuschkin, Christopher J. Anders, and Klaus-Robert Müller. Toward interpretable machine learning: Transparent deep neural networks and beyond. CoRR, abs/2003.07631, 2020. URL https://arxiv.org/abs/2003.07631. Yongsheng Shi, Dafne Campigli Di Giammartino, Derek Taylor, Ali Sarkeshik, William J. Rice, John R. Yates, Joachim Frank, and James L. Manley. Molecular architecture of the human pre-mRNA 3’ processing complex. Molecular Cell, 33(3):365–376, 2009. doi: 10.1016/j.molcel.2008.12.028. Avanti Shrikumar, Johnny Israeli, Karl Leswing, Peyton Greenside, Bharath Ramsundar, Michael Wainberg, and Benjamin Bond. simdna, 2019. URL https://github.com/kundajelab/simdna. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56): 1929–1958, 2014. URL http://jmlr.org/papers/v15/srivastava14a.html. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks, 2017. 24

Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, İlhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261–272, 2020. doi: 10.1038/s41592-019-0686-2. Martin Wainwright and Michael Jordan. Graphical models, exponential families, and variational inference. Foundations and Trends in Machine Learning, 1:1–305, 01 2008. doi: 10.1561/2200000001. Yeming Wen, Paul Vicol, Jimmy Ba, Dustin Tran, and Roger Grosse. Flipout: Efficient pseudo-independent weight perturbations on mini-batches, 2018. Jessica Zhou, Kaeli Rizzo, Ziqi Tang, and Peter K. Koo. Uncertainty-aware genomic deep learning with knowledge distillation. npj Artificial Intelligence, 2025. doi: 10.1038/s44387-025-00053-3. Preprint: bioRxiv 2024.11.13.623485.

25

Supplementary Materials for Uncertainty-Aware Deep Learning for Genomics Applications: Insights from an Empirical Study S1

Information for reproducibility

S1.1

Use case I: predicting protein binding from DNA/RNA sequence

This section provides additional reproducibility detail for the datasets introduced in Section 2 of the main text. We focus here on pre-processing, the derivation of the dataset settings, and the train/validation/test splits rather than repeating the high-level description. Scripts for data generation and pre-processing are provided in the code repository. S1.1.1

Real Dataset

The real dataset we used, was generated by PAR-CLIP experiments. These experiments can biochemically detect short sequences of RNA (i.e. binding sites), to which a specific known protein has bound. The data provided by these experiments is the list of binding site positions for a given RBP. In this work, we used a collection of existing PAR-CLIP datasets(Mukherjee et al., 2019) in the HEK293 cell line for three RBPs that are known to have strong sequence motifs: MBNL1, PUM2, and QKI. We selected these RBPs because our classifier performs well for proteins where there is a strong sequence preference for binding to be learned by the model. This helps us eliminate the uncertainty caused by lack of existence of a motif and enables us to build our comparison scheme for UQ in a more controlled setting. This data is publicly available and does not contain personal information. We followed the pre-processing steps from (Ghanbari & Ohler, 2020) and used the human GRCh37/hg19 reference for retrieving RNA sequence as the input for the classification task. The final dataset had onehot-encoded RNA sequence windows of size 150 nucleotide as the input and vectors of size three for the RBP classes as labels. The original data, was naturally multi-labeled, meaning there were instances of RNA sequence sections, where several RBPs would bind to different locations along this section. Also, the datasets had different coverage for different RBPs, therefore we had a different number of binding sites detected which caused the dataset to be strongly imbalanced (see Figure 1 in the main text). To generate single-label data settings, we simply removed samples where binding sites from several RBPs were present. To create the balanced data scenario, we sub-sampled larger classes to be in the size of the smallest class. As a result, we generated four dataset settings for the RBP-binding task: single-label balanced, single-label imbalanced, multi-label balanced, and multi-label imbalanced. To repeat our experiments across different random seeds, we repeated splitting each dataset into 65% training, 20% validation, and 15% test sets with 5 random seeds. S1.1.2

Simulated Dataset

To account for further dataset characteristics, we generated simulated datasets for a similar classification task to the RBP-binding, but this time for protein-binding events for DNA sequences. These proteins are called Transcription Factors (TFs) and similar to RBP, can influence the gene regulation process in the cell in various ways. The classification task remains the same: given a sequence of 150-nucleotide length, predict which proteins bind to it. The input data here is a section of DNA sequences and the target is a selection of Transcription Factors. We chose 6 TFs with widely known sequence motifs, namely, CTCF, HNF4A, JUN, MEF2A, MYC, and TAL1 as our targets for this classification task. We aimed to generate input sequences to create scenarios where the dataset has missing sequence motifs or the data is partially mislabeled. Both cases are common when working with noisy biological data. To this end, we used SimDNA(Shrikumar et al., 2019) software to synthesize DNA sequences and plant sequence motifs in them. SimDNA is a tool developed in Python for generating Genomic regulatory sequences. 26

Figure S1: Example of sequence motifs for selected RBPs (a) and TFs (b). Some RBPs and TFs have preferences in the sequence patterns (i.e. motifs) onto which they bind. Identifying these motifs can help predict protein binding activities in RNA or DNA. a) RBP motifs are extracted using XAI on the DeepRiPe model (Ghanbari & Ohler, 2020). b) TF sequence motifs are provided by JASPAR open-source database, version 2020 (Fornes et al., 2020).

It does so by first generating a background sequence and then embedding desired sequence motifs in the background. For our experiments, we have used the first-order Markov Model to generate DNA background sequences of length 150 and embedded a single motif sequence for a TF class. We used the Jaspar open-source database(Fornes et al., 2020) to obtain sequence motifs of our selected TFs. JASPAR provides manually curated TF binding profiles as position frequency matrices (PFMs). These PFMs are then used in SimDNA to embed a sequence motif of a TF into the background DNA sequence (see suppl. Figure S1). We generated three simulated datasets: a clean single-label balanced dataset as a noise-free reference, a noisy dataset where the sequences have no motifs, and a single-label balanced dataset where a portion of data is mislabeled. The simulated dataset consists of 2000 samples per class equally distributed to training, validation, and test sets (the entire dataset size is 12000). For the dataset with no motifs, we simply used the background sequences without implanting any motifs and for the mislabeled dataset, we swapped labels between the two TF classes of JUN and MEF2A for 30% of class samples. We repeat splitting each simulated dataset with 5 random seeds, similar to the real datasets. S1.1.3

Modeling specification

The base deterministic architecture is built based on DeepRiPe(Ghanbari & Ohler, 2020). This network has two one-dimensional convolutional layers, each followed by a max-pooling layer and a drop-out rate of 0.25. This is followed by two fully connected layers, the first one with 250 nodes and a drop-out rate of 0.25 and the last with 3 or 6 nodes, depending on the number of classes in the classification task. All layers except the last have ReLu for activation function. The deterministic model, as well as member models in ENS and MCD, have softmax activation for the output layer in the single-label dataset settings and the sigmoid function for multi-label. BNN doesn’t have an activation function for the output layer, but the respective function will be applied to the output in a later stage. The ENS is a collection of 10 deterministic models, each following the base model architecture. we initialized the member models with different random seeds and shuffled the training dataset with that seed. This is separate from the 5 random seeds used to create different data splits. For generating 100 iterations of inference on the test set, we repeat inference on each member model 10 times. For MCD, all three drop-out layers (two on convolutional layers and one on the dense layer) were activated at inference time. In BNN implementation, for all network layers, we used FlipOut layers from TensorFlow Probability which assumes a normal distribution over weights and biases. For training the BNNs we used mean-field variational 27

inference. The loss function for BNN is the sum of Kullback-Leibler (KL) Divergence loss and softmax cross-entropy loss with logits (for single-label) or sigmoid cross-entropy loss with logits (for multi-label) in TensorFlow. The loss function for deterministic, ENS, and MCD is just comprised of the cross-entropy loss.(Abadi et al., 2015) BNN requires a longer training time and a larger number of epochs to train enough to converge and reach comparable accuracy to the other models. Training BNNs in the multi-label RBP setting was unstable across random seeds, with some runs not converging or getting stuck in local minima; this improved when we reduced the learning rate and allowed longer patience for early stopping. We found BNNs sensitive to the choice of optimizer and hyper-parameters such as learning rate, whereas MC-dropout networks were robust to these choices, though the dropout rate and the choice of which layers to keep active at inference influence performance. Overall, the increased number of parameters and the approximate training of BNNs translate into a need for more computational resources and larger datasets. We set the maximum number of epochs in BNN training to 100 for single-label with early stopping with patience 50, and to 1000 maximum epochs and patience 100 for multi-label. For deterministic, ENS, and MCD maximum number of epochs was set to 50 with patience 5, except, for ENS training on simulated dataset with 6 classes, we needed longer training time to produce results comparable to BNN and MCD, therefore increased maximum number of epochs to 100 and patience to 10. Adam optimizer with batch size of 128 was used for training all the models, with a learning rate of 0.01 except for BNN in the multi-label setting, where we used 0.005. For generating distributions over predictions, the inference is repeated on the test set for 100 iterations, unless specified otherwise. The mean of the iterations is the prediction for the test samples and the standard deviation of the iterations is considered the uncertainty score. S1.2

Evaluation Metrics

In this section we introduce the metrics we used in our analysis for evaluating the predictive performance and the quality of uncertainty scores. We denote the dataset as D = {xn , yn }N n=1 where xn is the one-hot-encoded input sequence of length 150 (i.e. a 150 by 4 matrix) and yn is the one-hot encoded class label, y ∈ {1, ..., K}, where K = 3 for real dataset and K = 6 for simulated dataset. θ is the model parameters and pθ (y|x) is the predictive distribution that is produced by iterating inference on the model (MCMC sampling). µθ is the mean of the predictive distribution, used for evaluating predictive performance and σθ is the standard deviation of this distribution used as the uncertainty score. 1. Negative Log Likelihood (NLL) NLL is a widely-used metric for evaluating predictive uncertainty and is defined as : y − µθ (x) log σθ (x) − log pθ (yn |xn ) = + 2 2σθ (x)

2 (1)

2. Brier Score The Brier score is a measures of quality of the predictive uncertainty. It assigns a numerical score to a predictive distribution pθ (y|x), rewarding better calibrated predictions with a higher score (Lakshminarayanan et al., 2017; Gneiting & Raftery, 2007; Quinonero Candela et al., 2006). It’s equivalent to the squared error between the predictive probability of a label and one-hot encoding of the correct label in classification. ( K X
1 if k = y ∗ −1 ∗ ∗ 2 ∗ (2) BS = K tk − p(y = k|x ) with tk = 0 otherwise k=1 3. Kendall’s Tau Test This test measures correspondence between two rankings. Values close to 1 indicate strong agreement, and values close to -1 indicate strong disagreement.(Virtanen et al., 2020) The test provides p−values as well as the tau statistics. 28

4. Kolmogorov–Smirnov (KS) Test KS test is measure for comparing two samples in terms of goodness of fit(Hodges, 1958). It compares the underlying distributions of two independent samples that can be of different sizes and it does not make assumptions about the underlying distribution being a normal. Here we used a two-sided KS test with the method set to exact for comparing the distribution statistics. The null hypothesis of this test is that the two distributions are identical and the alternative is that the statistics provided by the test is the absolute difference between the empirical distribution functions of the samples. (Virtanen et al., 2020) We use this test to compare uncertainty scores over all samples in a test set, in different dataset settings. Meaning sample A is the list of σθ for test samples in scenario A and is compared against sample B for the same test set but in scenario B. An example is to compare a) distribution of uncertainty scores when a model is trained on balanced training data and is tested on balanced data against, b) the distribution of uncertainty scores when the same model is trained on imbalanced training data but tested on the same balanced test dataset. In such case, we hope that UQ model can show a shift in uncertainty scores of the smaller classes, reflecting on the imbalance of training data, therefore the KS test should reject the null hypothesis and measure the distribution shift in the uncertainty scores. Since this is a two-sided test, a large KS statistic show the distribution of uncertainty score has moved as a result of perturbation, but doesn’t say anything about the direction of the move. Moreover, we calculate the usual performance metrics for classification tasks, namely, accuracy, average precision, area under precision-recall curve (AUPR) and area under receiver operating characteristic curve (AUROC). Accuracy is computed per use case with the metric matching its labeling scheme: argmax agreement for single-label settings and element-wise agreement at a threshold of 0.5 for multi-label settings. Average precision is the mean over classes of the one-vs-rest average precision. S1.3

Further Experimental Details

We ran our experiments on an NVIDIA A40 GPU node with CentOS Linux 7 in our internal HPC cluster and are tracked using Neptune AI software (neptune.ai, 2024). Python version is 3.10 in Anaconda 24.3. Cuda version is 12.3 and cuDNN, 8.9, to be compatible with Tensorflow 12.6.1 and Tensorflow-Probability 0.24.0. For more information about packages and reproducibility please refer to the ReadMe file in the code repository. Each training experiment is repeated with 5 different random seeds. The seed is used to split the training, validation, and test sets, and to fix the random seed in relevant Python libraries. The results presented in this paper are produced with random seeds: 7, 5, 6, 9 and 11. When comparing experiments across different models (i.e. ENS, BNN, MCD), we compare results from the dataset splits on the same random seeds, to ensure that the comparison is done for models trained on the same training data and tested on the same test dataset. This is essential for comparing shifts in uncertainty scores or calculating correlations among predictions or among uncertainty scores for different models.

S2

Additional Results

S2.1

XAI and the TF simulation Strategies

We initially used SimDNA to generate simulated datasets for the same RBP-binding tasks. We generated RNA backgrounds and embedded motifs from RBPs. Then we ran our experiments and inspected the samples that were identified as highly uncertain by BNN, even in the clean single-label balanced setting with no noise. For the most uncertain test samples, we ran the Integrated Gradient (IG)(Sundararajan et al., 2017), an explainable AI (XAI) method, and looked into the attribution maps produced by IG on BNN. Using XAI together with UQ identified the source of uncertainty in those samples, which was due to a data simulation flaw. The attribution maps revealed that while we planted only a single motif in each input sequence, there was by chance a sequence motif of another protein present in the background (see Figure S2). This caused 29

labeled TARDBP

predicted QKI p=0.64 SD=0.310 <- followed the wrong motif TARDBP

0

20

40

60

QKI (not the label)

80

100

labeled RBFOX2

predicted QKI p=0.55 SD=0.290 <- followed the wrong motif

20

40

60

80

labeled ELAVL1

predicted MBNL1 p=0.56 SD=0.278 <- followed the wrong motif

0

20

40

60

140

QKI (not the label)

RBFOX2

0

120

100 ELAVL1

80

120

140

MBNL1 (not the label)

100

120

140

position in the 150 nt window

Figure S2: As our simulation strategy was not suitable for generating data for the RBP-binding task, we had several unwanted multi-labeled samples generated in the single-label setting. Having multiple class motifs is the reason for model confusion and higher uncertainty. This issue was resolved when we used the same simulation strategy for the TF-binding task. multiple class motifs to be present in the input which confused the model that was training in single-label setting. The reason for the simulation issue was that SimDNA is designed for synthesizing DNA sequences and implanting TF motifs, where TF motifs (10-15 nt) are usually longer than RBP motifs (typically 4-8 nt), therefore by chance, the probability of creating a background sequence with an RBP motif is higher (see suppl. Figure S1 for motif examples). This insight was the reason why we switched to using DNA sequences and TFs for the simulated data rather than RNA sequences and RBPs. This observation showed that XAI methods can be utilized for uncertainty-aware models to identify the source of uncertainty. S2.2

The 27-RBP application: extended analysis

This section collects the additional analyses of the 27-class RBP application introduced in Section 4.4: the precision-recall curves behind the filtering result, and three analyses of what drives the uncertainty on this task. S2.2.1

Precision-recall curves for the 27-RBP filtering experiment

The main text reports that filtering predictions on an uncertainty score below 0.1 raises average precision on the 27-class RBP task. Figure S3 shows the underlying precision-recall curves for 5 of the 27 classes, before and after filtering. S2.2.2

What drives the uncertainty on this task

The filtering experiment of Section 4.4 shows that the uncertainty score is useful, but not what drives it. We therefore examined the 27-class predictions further, using integrated gradients (Sundararajan et al., 2017) on the trained BNN and the structure of the model’s errors. Three findings are reported here. The accompanying 30

Figure S3: Precision-recall curves before and after uncertainty filtering (27-RBP task). Precisionrecall curves for 5 of the 27 classes for a BNN: a) unfiltered predictions; b) predictions filtered to an uncertainty score below 0.1 for the predicted class.

code repository contains the complete set of analyses, including further tests of additional candidate factors, which showed no other conclusive effect. Incorrect predictions carry no sequence-motif evidence For each test sequence we computed integrated-gradient attribution maps with respect to the predicted class, averaged over 20 draws from the BNN posterior, and summarized each map by the fraction of the total attribution falling inside the most informative 8-nucleotide window. RBP motifs are typically 4–8 nucleotides long (Gerstberger et al., 2014), so this window is matched to the scale of the feature being detected. As a reference for “no motif present” we repeated the procedure on dinucleotide-shuffled copies of the same sequences, which preserve base and dinucleotide composition while destroying any motif. Correctly classified sequences carry a clear motif signal: for MBNL1, PUM2 and QKI the attribution is concentrated +0.101, +0.070 and +0.026 above the shuffled reference (p < 10−4 in all three). Incorrectly classified sequences do not. Their attribution maps are statistically indistinguishable from, or below, the dinucleotide-shuffled reference in all three RBPs, and differ from the correctly classified ones with p between 10−22 and 10−34 . The model does not attend to a competing motif when it errs; it has no localiszd sequence evidence at all. Figures S4 and S5 show this for individual sequences. For each RBP we plot the attribution profile of one confidently correct, one correct but uncertain, and one confidently incorrect prediction, together with a dinucleotide-shuffled copy of the first. Beside each profile the most informative window is drawn as a logo in which letter height is the signed attribution of the base actually present, so the nucleotides the model used can be read directly. In the confidently correct sequences these are the published binding motifs: GCUU repeated three times for MBNL1, UGUA for PUM2 and UAAC for QKI, matching the motif panel in Figure S1a. In the confidently incorrect sequences no such motif is present and the attribution is spread across the window, at the level of the shuffled control. The sequences shown are the median of each group by attribution concentration rather than the sharpest examples available, so they are representative rather than best-case; their indices are given in the released code repository. The margin over the shuffled control tracks how well each class was learned: it is 0.104 for MBNL1, 0.070 for PUM2 and only 0.023 for QKI, the smallest of the three classes, whose motif is correspondingly the least distinct in the figure. Uncertainty is structured by protein complex membership The errors themselves are not arbitrary. We measured, for each ordered pair of RBPs, how often the model assigns a sequence of one class to the 31

attribution of the observed base known PUM2 motif: UGUA

correct, confident

PUM2 SD=0.000 p(PUM2)=1.00 conc.=0.212

correct, unsure

PUM2 SD=0.185 p(PUM2)=0.90 conc.=0.185

wrong, confident

PUM2 SD=0.007 p(FUS)=1.00 conc.=0.133

shuffled control

PUM2 dinucleotide-shuffled copy of the first sequence conc.=0.135

0

20

40

60

80

100

position in the 150 nt window

120

140

Figure S4: Attribution maps for individual PUM2 sequences. Rows: a confidently correct prediction, a correct but uncertain one, a confidently incorrect one, and a dinucleotide-shuffled copy of the first sequence. Left: attribution summed over the four bases at each position, with the most informative 8-nucleotide window shaded. Right: that window as a logo, letter height giving the signed attribution of the base present. The confidently correct sequence carries the UGUA motif; the confidently incorrect one carries no motif and is predicted as FUS. Sequences are the median of each group by attribution concentration.

32

attribution of the observed base known MBNL1 motif: GCUU / GCUUGC

correct, confident

MBNL1 SD=0.000 p(MBNL1)=1.00 conc.=0.253

correct, unsure

MBNL1 SD=0.194 p(MBNL1)=0.91 conc.=0.228

wrong, confident

MBNL1 SD=0.083 p(P53_NONO)=0.97 conc.=0.133

shuffled control

MBNL1 dinucleotide-shuffled copy of the first sequence conc.=0.149

0

20

40

60

80

100

position in the 150 nt window

120

140

attribution of the observed base known QKI motif: CUAAC

correct, confident

QKI SD=0.000 p(QKI)=1.00 conc.=0.156

correct, unsure

QKI SD=0.162 p(QKI)=0.94 conc.=0.164

wrong, confident

QKI SD=0.001 p(NOP56)=1.00 conc.=0.150

shuffled control

QKI dinucleotide-shuffled copy of the first sequence conc.=0.133

0

20

40

60

80

100

position in the 150 nt window

120

140

Figure S5: Attribution maps for individual MBNL1 and QKI sequences. As Figure S4, for the other two RBPs. The confidently correct MBNL1 sequence carries three copies of GCUU; the confidently correct QKI sequence carries UAAC. The confidently incorrect predictions are assigned to P53_NONO and to NOP56 respectively, the latter being the class QKI is most often confused with (Figure S6). 33

a)

EIF3G→EIF3D EIF3A→EIF3D EIF3D→EIF3A EIF3G→EIF3A EIF3A→EIF3G FUS→TAF15 CPSF3→CPSF4 NOP58→FBL TAF15→FUS CPSF4→CPSF3 FBL→NOP58 LIN28A→LIN28B NOP56→NOP58 NOP58→NOP56 LIN28B→LIN28A NOP56→FBL FBL→NOP56 EIF3D→EIF3G

101

confusion enrichment (observed / expected)

b)

p = 8.1e-05

100

10−1 different complex

same complex

eIF3 complex FET family CPSF complex box C/D snoRNP LIN28 paralogs

0

2

4

6

8

10

12

14

16

confusion enrichment (observed / expected)

Figure S6: Misclassifications concentrate within protein complexes. a) Confusion enrichment, defined as the observed rate of predicting class b for sequences of class a divided by the rate expected from how often b is predicted overall. Points are ordered RBP pairs; a value of 1 (dashed line) is the rate expected by chance. b) The individual pairs within each complex or family. Enrichment is a ratio, so panel a) uses a logarithmic axis.

other, relative to how often that second class is predicted overall. This normalization matters: without it a class that is simply predicted frequently would appear to attract errors specifically. Pairs of RBPs belonging to the same protein complex or family are confused 2.97 times more often than expected, against 0.78 for pairs from different complexes (Mann–Whitney, p = 8 × 10−5 ; Figure S6). The effect is reciprocal in both directions for the subunits of eIF3 (up to 16-fold), for the FET-family proteins FUS and TAF15 (3–4-fold), and for the CPSF subunits (3–4-fold). These groupings are established independently of our data: the eIF3 subunits A, D and G bind an overlapping program of messenger RNAs (Lee et al., 2015), the FET proteins show closely similar binding profiles in PAR-CLIP (Hoell et al., 2011), and CPSF3 and CPSF4 are subunits of one pre-mRNA 3′ processing complex (Shi et al., 2009). The interpretation is that the model is least certain where the underlying biology is genuinely ambiguous. A transcript bound by one member of a complex is frequently bound by the others, so the single-label ground truth is partly arbitrary for those sequences, and a well-behaved uncertainty score should be elevated on them. We tested if we can associate high uncertainty score to presence of the wrong/swapped motifs for these misclassifications but couldn’t establish that connection. This is expected, since these proteins are general initiation factors that are not known to bind to a specific sequence motif, and will target different messenger RNAs. Therefore XAI on the model doesn’t pick any motifs for these protein complexes.

Class size drives the uncertainty score The third factor is not a property of individual sequences but of the classes themselves, and is reported in Section 4.4. The effect depends on how the scores are grouped. Grouping by the class the model predicted, which is what the filtering threshold acts on, the Spearman correlation between training-set size and median uncertainty is ρ = −0.59 (p = 0.001, n = 27) and the five smallest classes have a median 57 times that of the five largest. In the controlled experiments we construct the imbalance deliberately; here it is simply a property of how many binding sites each protein has in the PAR-CLIP data, and the BNN’s uncertainty tracks it. It is also worth mentioning that the mechanism behind the filtering result of Section 4.4: discarding high-uncertainty predictions removes disproportionately from the classes the model had least opportunity to learn, which is why average precision improves most for the rarest RBPs. 34

MCD

BNN

ENS

CTCF

HNF4A

JUN

MEF2A

MYC

TAL1 −80

−60

−40

−20

−40

log2(SD)

−20

0

log2(SD) balanced, clean

−40

−20

0

log2(SD)

balanced, missing motif

Figure S7: Sensitivity of uncertainty scores to missing motifs (simulated TF data). Distribution of uncertainty scores when the model is trained and tested on clean balanced single-label data (green) versus background sequences with no planted motif (orange). Rows are the six transcription factors; columns are MCD, BNN, and ENS. Scores are on a log2 scale. Results are shown for random seed 7. S2.3

Missing-motif noise on the simulated TF data

The main text reports the mislabeling perturbation, where the three UQ methods differ in how specifically the uncertainty increase is confined to the corrupted classes. The second perturbation we applied to the simulated TF data replaces the input with background sequences carrying no planted motif, removing the predictive signal altogether. Figure S7 shows the result. Every class shifts by a large and roughly equal amount — between 8 and 16 log2 units — and this is true for all three methods, so the experiment does not separate them. We report it as a positive control: it confirms that the shift measurement responds as expected when the signal is unambiguously destroyed, which is what makes the more selective response to mislabeling in the main text interpretable. S2.4

Uncertainty scores on the raw scale

The sensitivity figures in the main text plot uncertainty scores on a log2 scale. Figures S8 and S9 give the same data on the raw scale, for reference. They illustrate why the log scale is necessary rather than cosmetic: MC-dropout’s predictive standard deviations reach only about 10−6 on the simulated TF task, and those of BNN and ENS about 0.4, so on a linear axis every class in every panel collapses to a single spike at the origin and neither the size nor the direction of the shift can be read off. S2.5

Uncertainty shift in class imbalance

The sensitivity figures in the main text show, for one random seed, that the distribution of uncertainty scores moves when the training set is made imbalanced. Figure S10 the change in training-set size against the median log2 (SD) shift for each class separately, with one point per random seed. There are two different 35

a)

MCD

BNN

ENS

CTCF

HNF4A

JUN

MEF2A

MYC

TAL1 0.0

0.5

1.0

1.5

0.0

1e−6

SD

0.2

0.3

0.4

0.0

SD

balanced, clean

b)

0.1

0.2

0.4

SD

balanced, mislabelled

MCD

BNN

ENS

CTCF

HNF4A

JUN

MEF2A

MYC

TAL1 0.0

0.5

SD

1.0

1.5

0.0

1e−6

0.2

0.4

SD

balanced, clean

0.0

0.2

0.4

SD

balanced, missing motif

Figure S8: Sensitivity of uncertainty scores on the raw scale (simulated TF data). Predictive standard deviation on a linear axis, for a) partially mislabeled noise and b) missing-motif noise. Note the ×10−6 multiplier on the MC-dropout axes.

36

a)

MCD

BNN

ENS

MBNL1

PUM2

QKI

0.0

0.5

1.0

1.5

0.0

1e−6

SD

0.2

0.3

0.4

SD

balanced, single-label

b)

0.1

0.0

0.1

0.2

0.3

SD

imbalanced, single-label

MCD

BNN

ENS

1.5

0.000 0.025 0.050 0.075

0.00 0.05 0.10 0.15 0.20

1e−6

SD

SD

CD14+ Mono

CD4+ T activated

Erythroblast

NK

0.0

0.5

SD

1.0

balanced, single-label

imbalanced, single-label

Figure S9: Sensitivity of uncertainty scores to class imbalance on the raw scale. Predictive standard deviation on a linear axis, for a) the real RBP use case and b) the single-cell use case. Green: balanced training set; orange: highly imbalanced training set.

37

Real RBP median log2 (SD) shift

MC-dropout 5

ρ = +0.50

Deep ensemble 0

ρ = +1.00

7.5

ρ = +0.50

−2

5.0 0

−6

0.0

−5

10

−4

2.5

QKI ×0.99

Single-cell median log2 (SD) shift

BNN 10.0

PUM2 ×2.49

MBNL1 ×8.86

QKI ×0.99

4

ρ = +0.20

PUM2 ×2.49

MBNL1 ×8.86

0.5

ρ = -0.80

PUM2 ×2.49

MBNL1 ×8.86

ρ = +0.40

0.0

2

5

QKI ×0.99

−0.5

0

0

−1.0

−2

−1.5

−5 −4 Eryth. ×0.22

CD4+ T ×1.23

NK ×1.58

CD14+ ×2.35

Eryth. ×0.22

CD4+ T ×1.23

NK ×1.58

CD14+ ×2.35

Eryth. ×0.22

CD4+ T ×1.23

NK ×1.58

CD14+ ×2.35

training-set class size, imbalanced relative to balanced, in rank order (left = depleted, right = enriched)

Figure S10: Change in training-set class size against the shift in uncertainty. Each point is one class in one of 5 random-seed repeats. Classes are ordered by how their training-set size changed when imbalance was introduced, given below each label as a multiplier; left-most is the most depleted. Top row: real RBP use case, evaluated on a class-balanced test set. Bottom row: single-cell use case; abbreviated cell types are Erythroblast, CD4+ T activated, NK and CD14+ Mono. The dashed line is a least-squares fit and ρ is the median Spearman correlation across seeds. A negative ρ means the classes that lost training data became the more uncertain ones.

questions to be asked about the class imbalance: does the model produce higher uncertainty scores overall for all classes when imbalanced is introduced to data? and does the model produce higher uncertainty score for the smallest class compared to the other classes in the imbalanced setting? Answering the first question will be affected by how we built the balance/imbalanced versions of the datasets for our experiments. For commenting on usability of UQ methods in working with highly imbalanced genomics datasets, we care about the answer to the second question. The two use cases construct imbalance differently: in the single-cell data the minority class is depleted from the balanced version: Erythroblast falls from 2673 to 587 training cells, a reduction of 78%, while the other three classes grow. In the RBP data, the imbalanced version one the real starting point and the the smallest class keeps essentially all of its data (QKI, 816 to 808 sequences) and the balanced version is a subset of this data. Only the single-cell panels therefore test whether the harshly sub-sampled class becomes the most uncertain one; the RBP panels test the complementary manipulation. On the single-cell data BNN gives a Spearman correlation of −0.80 between the change in class size and the shift in uncertainty, and the correlation is negative in every one of the five seeds (−0.80, −1.00, −0.80, −0.80, −0.60). The relationship therefore does not depend on which seed is shown, or on excluding any seed. MC-dropout (+0.20) and the deep ensemble (+0.40) have the opposite sign: for both, the classes that gained data are the ones whose uncertainty rises most. On the RBP data all three methods are positive, as expected when nothing was depleted. 38

S2.6

Degenerate MC-dropout uncertainty scores

MC-dropout’s predictive standard deviation is identically zero for a large fraction of test samples: 2% on balanced single-label single-cell data, 28% on the imbalanced version, 40–51% across the single-label RBP settings, and up to 68% in the worst case. The softmax saturates and every Monte Carlo pass returns the same probability vector to floating-point precision. Dropout is active in all runs and the variance is non-zero somewhere in every one of them, so these are genuine model outputs rather than an implementation error, but for the affected samples MCD provides no ranking. This inflates its apparent agreement with the other methods. Restricting Kendall’s tau to samples where the MCD score is non-zero removes most of it: on imbalanced single-label single-cell data the median τ against ENS falls from +0.36 to +0.02, and on the mislabeled TF data it changes sign, from +0.16 to −0.13. MCD separates saturated from non-saturated samples but ranks nothing within either group, and because the size of the saturated block varies from 2% to 68% its tau values are not comparable across settings. The corresponding values for BNN and ENS are unaffected. S2.7

Detecting out-of-distribution sequences

A common situation in genomics is applying a model trained on human data to a novel, out-of-distribution domain, such as a newly sequenced organism. Understanding the binding interactions of RNA-binding proteins in viruses such as SARS-CoV-2 can provide valuable insights into the infection mechanism. RBP binding datasets generated by experimental assays are widely available for the human transcriptome, yet similarly rich datasets are missing for novel viral genomes. At the same time, the differences in composition between the viral and the human genome limit the effectiveness of applying a model trained on one genome’s data to the other: the viral sequence composition is biased towards AT relative to the coding and non-coding human transcriptome (Horlacher et al., 2023). More generally, novel or rapidly emerging organisms are precisely the case where labeled data are expensive or impossible to generate quickly, as during the SARS-CoV-2 pandemic, and where the underlying biology is not yet fully understood, so the extent to which a model trained on human data can be expected to transfer is not known in advance, even between datasets from the same experimental protocol and tissue. We therefore treat the viral data as an out-of-distribution case for models trained on human data and ask whether the uncertainty scores flag it as such. No labels exist for the viral windows, so the evaluation rests entirely on the uncertainty scores — precisely the label-free setting this framework is designed for. We trained on human single-label RBP data and tested on eight human pathogenic coronavirus genomes, tiled into windows of the same length as the training sequences. The models are those of Section S1.1.1, applied without retraining. All UQ methods show a shift towards higher uncertainty on the out-of-distribution data, with BNN reflecting this most clearly (Figure 11), indicating that the uncertainty score can flag inputs that fall outside the training distribution even without labels. Quantifying the shift with the two-sided KS statistic between the human and viral score distributions, over 5 random seeds and all eight genomes, gives a median of 0.62 for BNN, 0.56 for ENS and 0.43 for MCD. A more practical framing is to set a flagging threshold at the 95th percentile of the human scores, as a practitioner without viral labels would have to: BNN then flags 36% of viral windows as unusually uncertain and ENS 27%, whereas MCD flags only 7% — barely above the 5% that the definition of the threshold guarantees by construction. On this task MCD’s uncertainty score is therefore of little practical use for detecting that the model has left its training distribution. This extends the observation that BNN better reflects class imbalance to a second, quite different kind of dataset shortcoming. S2.8

MNIST as a benchmark domain

We additionally use the MNIST dataset as a benchmark to compare the performance of BNN and MCD in different scenarios common in genomics applications, namely imbalanced and scarce data. The MNIST classification task (LeCun et al., 1998) is a well-established benchmark in machine learning. The dataset contains 70,000 images of handwritten digits in grayscale. The dataset is normalized and balanced, with each digit centered in a 28 × 28 pixel box. We run several experiments on this multi-class classification problem 39

Figure S11: Class-size distribution of the imbalanced MNIST dataset. Number of samples per digit class in the sub-sampled, imbalanced version of MNIST. as a benchmark for comparing MC-dropout and the BNN in terms of the prediction performance and the estimated uncertainty. To further evaluate these approaches, we repeat the experiments on an imbalanced version of the MNIST dataset with around 42,000 samples. The imbalanced version of the dataset is generated by sub-sampling from the original dataset using the following fractions: class 0: 0.5, class 1: 0.7, class 2: 0.8, class 3: 0.3, class 4: 0.6, class 5: 0.8, class 6: 0.4, class 7: 0.2, class 8: 0.9 and class 9: 0.8 (see Figure S11). Figure S12 illustrates the overlap between the most uncertain samples identified by BNN and MCD in the MNIST classification task, mirroring the agreement analysis on the genomics data. By visual inspection of the most uncertain samples per class, we find the uncertain samples identified by MCD to be more similar to what we would consider an ambiguous case. The picture matches the genomics results in two respects. The overlap between the two methods’ most uncertain sets is well above chance but far from complete, and it varies substantially between digit classes and between the balanced and imbalanced settings; in the imbalanced case the spread across classes widens considerably, with some digits agreeing on more than 70 of the top 10% and others on fewer than 20. Figure S13 shows the samples themselves. Disagreement between the two methods therefore does not isolate a subset of “genuinely” hard images; it reflects their different posterior approximations, exactly as argued for the genomics tasks in Section 4.1 of the main text. These MNIST experiments are intended as a qualitative cross-check outside genomics rather than as a third use case. On MNIST we compare only BNN and MC-dropout, and none of the conclusions in the main text depend on them.

40

Figure S12: Overlap of the most uncertain MNIST samples identified by BNN and MCD. Number of samples considered uncertain by both the BNN and MCD models in the MNIST task in two scenarios: (a) balanced dataset and (b) imbalanced dataset. The first row shows the count of common uncertain samples between the top 50 most uncertain samples for each class as indicated by the BNN compared to those indicated by MCD. The second row shows the count of common top 10% most uncertain samples.

41

Figure S13: MNIST samples identified as most uncertain. Columns are digit classes 0–9. Top block: samples among the most uncertain for the BNN; bottom block: samples among the most uncertain for MC-dropout.

42

Related documents

• Quality-Driven Selective Mutation for Deep Learning#134539
• Possibilistic Predictive Uncertainty for Deep Learning#151751
• Should All Noises Be Treated Equally: Impact of Input Noise Variability on Neural Network Robustness#1084451
• A survey on deep learning tools dealing with data scarcity: definitions, challenges, solutions, tips, and applications#199544
• Uncertainty quantification for trustworthy deep learning: Methods and measures#414125