Measuring consistency via ensemble margin and local prediction variability: Auditing decision systems in the presence of predictive multiplicity

Measuring consistency via ensemble margin and local prediction variability: Auditing decision systems in the presence of predictive multiplicity Sinjini Banerjee1 , Tim Marrinan2 , Anand D. Sarwate1 Department of Electrical and Computer Engineering, Rutgers University 2 Pacific Northwest National Lab sb1977@rutgers.edu, timothy.marrinan@pnnl.gov, anand.sarwate@rutgers.edu

arXiv:2609.01397v1 [stat.ML] 1 Sep 2026

1

September 2, 2026

Abstract The Rashomon effect is a machine learning phenomenon where equally accurate models produce different predictions for the same inputs (predictive multiplicity). Existing work primarily focuses on multiplicity within individual models, but in more complex decision systems, the impact of the Rashomon effect is less well understood. In this work, we study multiplicity from the perspective of auditing incorrect ensemble predictions, where the decision to divert an instance for human review is based on a consistency criterion that combines the ensemble margin with a measure of local prediction variability for each constituent model. With mild assumptions about stability and smoothness, we show that the consistency scores of finite ensembles converge to the corresponding consistency score of the expected model from the Rashomon set as the ensemble size and the number of samples used to measure local prediction variability increase. To demonstrate the efficacy of the proposed criterion, we evaluate the framework with respect to transformer models applied to natural language understanding tasks and parameter-efficient fine-tuning of large language models used for tabular data classification tasks. Our experiments show that ensembling models from the Rashomon set substantially reduces the risk of incorrect predictions going unchecked compared with auditing a single model, while incurring only a moderate increase in the number of diversions. Moreover, the auditing behavior of the full Rashomon set can be closely approximated by finite ensembles of relatively modest size, with the risk approaching zero for some datasets. We further demonstrate that the proposed measure exhibits stronger agreement with established predictive multiplicity metrics than existing consistency measures, providing a more reliable way to capture multiplicity in the Rashomon set.

1

Introduction

Centralized machine learning marketplaces have become a critical component of the modern AI ecosystem, providing practitioners with access to a wide range of pre-trained models for diverse applications. Platforms such as Hugging Face Hub (Wolf et al., 2020), AWS Marketplace (Amazon Web Services, 2026), and NVIDIA NGC (NVIDIA Corporation, 2026) host thousands of models, allowing users to rapidly deploy state-of-the-art systems without training models from scratch. An underlying assumption in model selection from these marketplaces is often that selecting the “best” model for a task is straightforward, where “best” is defined by performance metrics such as accuracy or F1-score. This assumption overlooks a fundamental property, known as the Rashomon effect, which describes the existence of multiple equally well-performing models achieving different predictions on a held-out test set (Breiman, 2001). In over-parameterized networks, the “black box” nature of these models makes it difficult to examine the flow of information through their billions of parameters and interpret their decisions. This lack of transparency can lead to low trust and confidence in these systems, creating challenges for their 1

deployment in safety-critical domains such as healthcare (Peterson, 2019; He et al., 2019), and autonomous vehicles (Perez-Cerrolaza et al., 2024). From a data-centric viewpoint, the Rashomon effect leads to a phenomenon known as predictive multiplicity, where competing models can disagree on individual test instances (Marx et al., 2020). In high-stakes domains such as healthcare, finance, and criminal justice, this instability can lead to inconsistent diagnoses, credit approvals, or risk assessments, undermining both fairness and accountability (Dai et al., 2025). These costs motivate the need for tools that identify when predictions are unstable and should be treated with caution. One way to mitigate predictive multiplicity is through ensembling. By aggregating predictions through multiple models, ensembles are expected to reduce variance, and, in the limit of large ensemble sizes, converge to the expected prediction under the model distribution. Deep ensembles are widely used in practice to improve accuracy and estimate uncertainty (Lakshminarayanan et al., 2017; Fort et al., 2019; Ganaie et al., 2022; Rahaman & Thiery, 2021). In practical scenarios, ensembles are often small, so predictive inconsistency may persist across independently sampled ensembles, even when the constituent models in each ensemble are individually near-optimal. This paper focuses on understanding and mitigating ensemble inconsistency arising from predictive multiplicity in small ensembles. While prior work has studied multiplicity across individual models, there is limited understanding of instance-level consistency guarantees for finite ensembles drawn from a Rashomon set. Moreover, consistency does not imply correctness: models may agree yet still be wrong, leading to missed diversions of erroneous predictions. We take auditing with diversion as our motivating application: instances can either be classified by the model or be diverted to an expert. Such systems must balance risk (the model’s error) against the cost of using the expert. We show that appropriately leveraging ensembles can reduce missed diversions and improve decision reliability.

1.1

Auditing and diversion to human experts

Modern machine learning models are abundant; in many cases, they achieve state-of-the-art performance on complex tasks. However, even the best-performing models may eventually make mistakes on unseen data. Suppose that there exists an idealized baseline model, fbase , that is the “gold-standard” for a given deployment scenario. Despite being the best model, fbase will inevitably make a prediction that is contradicted by domain experts. To reduce the risk of making bad decisions due to model errors, we would like to implement an automated “auditing” system to flag untrustworthy predictions and divert those decisions to a human expert for review. Human intervention ensures that edge cases and instances where the model is uncertain are handled using expert judgment. However, deciding when to intervene can be difficult, and mistakes can be costly. Failing to intervene when the baseline model makes an incorrect prediction can harm people whose lives are affected by the decisions, but deferring to an expert is also expensive and may be infeasible given the number of available experts. Since ground-truth labels are typically not available in practice, proxy measures derived from the model may help us decide when to divert. We consider a binary classification example, where x ∈ Rd has label y ∈ {−1, +1}. Let f : Rd → [0, 1] be a predictive model
such that f (x) is the estimated probability that x has label y = +1, and let 1 h(x; f ) = 1 f (x) ≥ 2 be an associated “hard” decision threshold for classifying x. The error of using f to predict y from x can be written as e(x; f ) = 1(h(x, f ) ̸= y). Now suppose we have an automated auditing system, r : F × Rd → {0, 1}, that takes in a function, f ∈ F, and an input instance, x ∈ Rd , and returns a risk assessment for the prediction, f (x). The auditing system returns r(x; f ) = 0 if the prediction made by the model is trustworthy, and it returns r(x; f ) = 1 if the case should be diverted to a domain expert for review, where we assume it can be correctly labeled by the reviewer. Discrepancies between e(x; f ) and r(x; f ) result in unnecessary costs to the system. When e(x; f ) = 0 but r(x; f ) = 1, the auditing system has diverted a case unnecessarily, creating extra work for the reviewers. We refer to this as an “audit cost”. In contrast, when e(x; f ) = 1 but r(x; f ) = 0, the auditing system fails to flag an incorrect prediction, creating potential downstream consequences. We

2

refer to this scenario as an “audit risk.” Audit risks and audit costs can be considered as errors of the auditing system itself. Let us return to our idealized baseline model, fbase . It may not be feasible to deploy the gold-standard model in all scenarios due to cost, resource availability, or other constraints. In this case, we may need to deploy a cheaper alternative, which we will refer to as the candidate model, f. Regardless of how well f approximates fbase , both systems are liable to make mistakes on some test instances. Thus, the auditing system r is used to reduce the risk of errors. The central problem of this work is how to design an auditing system, r, and a candidate function, f , such that the same set of samples is diverted for human review, whether we use fbase or f . In other words, we want r(x; f ) ≈ r(x; fbase ), while minimizing the excess audit costs and audit risks. Comparing the behavior of an auditing system applied to the candidate model and the baseline model gives rise to 16 possible outcomes, corresponding to all combinations of (e(x; f ), r(x; f )) for the candidate and (e(x; fbase ), r(x; fbase )) for the baseline. Table 1 organizes these outcomes in a 4×4 matrix, where each row corresponds to one candidate outcome and each column corresponds to one baseline outcome. Each cell summarizes the difference in audit cost and audit risk for the candidate model relative to the baseline. The most desirable outcomes are those in which the candidate meets the baseline, meaning that it reaches the same auditing decision as the baseline. These include cases where both systems correctly automate a prediction, both correctly divert it, or both incur the same unavoidable audit cost or audit risk. In Table 1, these outcomes are represented by the diagonal cells. We refer to these costs and risks as unavoidable because they arise from the behavior of the baseline model itself. For example, if a baseline prediction is diverted that was ultimately determined to be correct, or an erroneous baseline prediction is not diverted, the resulting audit risk and cost cannot be eliminated. Disagreements between the two systems instead introduce excess audit cost, excess audit risk, or a mixed outcome in which one quantity improves at the expense of the other. Some of these disagreements correspond to reduced audit cost or reduced audit risk, where the candidate performs better than the baseline on a particular instance. The objective of an auditing strategy is to minimize excess costs and risks while matching the baseline as closely as possible. In our experiments in Section 4.2.3 and Appendix A.7.3, we observe that the candidate model typically achieves a lower audit cost than the baseline, but only at the expense of a higher audit risk.

1.2

Our contributions

We develop a framework for balancing the costs and risks of diverting to human experts by using an ensemble of candidate models within an auditing system. To do this, we leverage the framework of predictive multiplicity and the Rashomon set of models. Prior work has largely focused on disagreement between individual models. We investigate how multiplicity propagates to downstream auditing systems and how finite ensembles can be used to improve reliability. Our baseline model fbase is the expected model over the Rashomon distribution, while a candidate model f is an ensemble of finitely many models drawn from the Rashomon set. We can use this to assess how ensemble size, risks, and costs are related in auditing systems. We call this choice of baseline a warehouse model and the finite ensemble a retail model as a metaphor for how the two ensembles are related. Our contributions are listed below. • We propose a new consistency measure, (β, σ)-consistency, that uses ensemble margin information and local perturbation stability in the embedding space of an input. This differs from prior measures of multiplicity that analyze a single candidate model. Sampling models allow us to study how well the retail set approximated the warehouse model by reducing variance induced by predictive multiplicity. • Under stability and local Lipschitz assumptions, we provide probabilistic guarantees showing that if an instance has a high (β, σ)-consistency score with respect to the baseline model, then finite retail 3

Table 1: Matrix of auditing outcomes comparing the audit system of a candidate, f , with the audit system of a baseline, fbase . Rows correspond to the candidate outcome (e(x; f ), r(x; f )) and columns correspond to the baseline outcome (e(x; fbase ), r(x; fbase )). Each cell summarizes how the candidate changes audit risk and audit cost relative to the baseline. Green cells indicate agreement with the baseline, red cells indicate increased audit risk relative to baseline, light pink cells indicate reduced audit risk relative to the baseline, yellow cells indicate increased audit cost relative to the baseline, and light yellow cells indicate reduced audit cost relative to baseline. Neutral outcomes in gray cells indicate that baseline and candidate models disagreed on their predicted labels but both correctly diverted or correctly automated the instances, and purple cells indicate mixed outcomes when baseline and candidate models disagreed on their predicted labels but both incorrectly diverted or incorrectly automated the instances. Baseline model (fbase ) - (e(x; fbase ), r(x; fbase )) Candidate model (f ) (e(x; f ), r(x; f ))

(0, 0)

(0, 1)

(1, 0)

(1, 1)

(0, 0)

Outcome 0 Baseline

Outcome 4 Cost ↓

Outcome 12 Neutral

(0, 1)

Outcome 1 Cost ↑

Outcome 5 Baseline

Outcome 8 Risk ↓ Outcome 9 Risk ↓ Cost ↑

(1, 0)

Outcome 2 Risk ↑

Outcome 6 Risk ↑ Cost ↓

Outcome 10 Baseline

Outcome 14 Risk ↑

(1, 1)

Outcome 3 Neutral

Outcome 7 Cost ↓

Outcome 11 Risk ↓

Outcome 15 Baseline

Outcome 13 Cost ↑

set ensembles will also assign high consistency scores to these instances with probability increasing exponentially in both the ensemble size M and the number of perturbation samples S. • We operationalize (β, σ)-consistency in an auditing application in which we threshold consistency scores to decide whether predictions should be diverted for expert review. • We empirically validate our results on fine-tuning applications with large language models. Our results quantify how increasing the number of models in the ensemble reduces predictive multiplicity and audit cost variability, while decreasing audit risk.

1.3

Related work

Our paper incorporates ideas from several different bodies of prior work. Learning-to-reject and learning-to-defer. In the selective classification or learning-to-reject literature (Herbei & Wegkamp, 2006; Bartlett & Wegkamp, 2008; Yuan & Wegkamp, 2010; Feng et al., 2023), a classifier may abstain from making predictions on instances that are considered difficult or unreliable. Learning-to-defer extends this framework by explicitly incorporating the predictions or expertise of the downstream decision-maker. In many selective classification systems, the decision to reject a prediction is often implemented by thresholding confidence or uncertainty-based measures. Common choices include maximum softmax probability, which captures confidence in the top prediction, the margin between the top two classes, and entropy, which reflects overall uncertainty in the predictive distribution (see recent

4

surveys by Zhang et al. (2023) and Hendrickx et al. (2024)). Applying a threshold to these measures determines if the instance is diverted. Rashomon sets and Predictive Multiplicity. Modern machine learning models, particularly deep neural networks, operate in overparameterized regimes where many distinct solutions achieve nearly identical empirical performance. This Rashomon effect, first described by Breiman (2001)1 implies that empirical risk minimization yields a set of near-optimal models rather than a unique predictor, formalized as the Rashomon set (Fisher et al., 2019; Semenova et al., 2022; Ciaperoni et al., 2024; Xin et al., 2022). Although these models are indistinguishable under aggregate metrics, they can produce different predictions or confidence scores on individual inputs, leading to predictive multiplicity. This phenomenon has been widely studied in classification settings (Marx et al., 2020; Watson-Daniels et al., 2023; Black et al., 2022a; Hsu et al., 2024; Zhu et al., 2024), including its dependence on data complexity (Cavus & Biecek, 2025) and information-theoretic characterizations (Hsu & Calmon, 2022). Predictive multiplicity arises even with identical architectures and data due to non-convex optimization and training stochasticity (Gundersen et al., 2023), with factors such as random initialization leading to substantially different learned functions despite similar accuracy (Banerjee et al., 2025; Summers & Dinneen, 2021; Jordan, 2023; Bouthillier et al., 2021; Picard, 2021; Henderson et al., 2018; Melis et al., 2018; Reimers & Gurevych, 2017; Dodge et al., 2020). In high-stakes settings, such variability can lead to inconsistent decisions, undermining reliability and trust. Consistency Analysis in Multiplicity. In the multiplicity literature, consistency across models has been used to characterize the stability of individual predictions. Consistency has been studied through model agreement (Marx et al., 2020; Black et al., 2022b), probabilistic prediction variance across the Rashomon set (Hsu & Calmon, 2022; Watson-Daniels et al., 2023), stochastic perturbations such as dropout (Hsu et al., 2024), invariance of counterfactual explanations (Black et al., 2022a; Hamman et al., 2023; Leofante et al., 2023), and stability under fine-tuning variability in large language models (Hamman et al., 2025). Some of the works provide probabilistic guarantees on a consistency measure, ensuring that one model’s decision will not be invalidated by another. A key insight from these works is that consistency depends not only on confidence but also on local stability. Predictions far from the decision boundary and robust to small input perturbations are more likely to remain invariant across models. Local perturbation analysis. Local perturbation analysis has been widely used in the machine learning literature as a principled way to probe and quantify robustness and consistency of machine learning models, especially in over-parametrized settings. For instance, randomized smoothing yields classifiers with provable local robustness guarantees (Cohen et al., 2019), while combining smoothing with ensembles improves stability under perturbations (Horváth et al., 2022). Similarly, in the multiplicity literature, stability of counterfactual explanations depends on local smoothness (Black et al., 2022b), and Lipschitz-based analyses provide probabilistic guarantees for prediction invariance under model perturbations (Hamman et al., 2023). Sampling local neighborhoods in representation space further enables efficient estimation of agreement across fine-tuned models (Hamman et al., 2025). Despite their promise, existing consistency measures in the multiplicity literature have important limitations. First, many approaches rely on hard labels, such as model disagreement, discarding richer information contained in predictive confidence. Second, while these methods are effective at identifying consistent test instances, they can misclassify inconsistent instances as consistent, leading to high missed diversions. More fundamentally, consistency does not imply correctness: multiple models may agree on a prediction and still be jointly wrong. As a result, consistency-based approaches may fail to divert 1

Named after the movie Rashōmon, directed by Akira Kurosawa and based on the short story In a Grove by Ryūnosuke Akutagawa.

5

erroneous predictions, which is particularly problematic in high-stakes settings where missed errors carry significant risk. Ensembling. A widely adopted approach for mitigating predictive multiplicity is ensembling, in which predictions from multiple models are aggregated, typically by averaging their outputs. The core idea behind deep ensembling is that a group of diverse baseline predictors often outperforms a single predictor by reducing variance, mitigating overfitting, and capturing multiple modes of the hypothesis space. Lakshminarayanan et al. popularized deep ensembles as a scalable alternative to Bayesian neural networks, showing that ensembles of independently trained models provide strong empirical uncertainty estimates and improved robustness to distribution shift (Lakshminarayanan et al., 2017). Subsequent work has analyzed deep ensembles from a loss-landscape perspective, demonstrating that random initialization leads to convergence to distinct modes that are effectively exploited through the weighted averaging of predictions (Fort et al., 2019). Empirically, deep ensembles differing only in random seed have been shown to perform competitively across vision and language tasks (Lakshminarayanan et al., 2017; Lee et al., 2015), while alternative ensemble construction methods such as bootstrapping may not offer additional benefits in deep learning settings (Nixon et al., 2020). Ensembles have also been studied as variance-reduction tools in randomized smoothing frameworks (Horváth et al., 2022), further reinforcing their role in reducing predictive multiplicity. Ensembles of fine-tuned language models have also been shown to improve predictive performance and uncertainty estimation (Wang et al., 2023; Pineda Arango et al., 2024). Despite the benefits of ensembling, sampling a small number of models from the Rashomon set can introduce significant Monte Carlo error in the ensemble’s estimate of the expected prediction. As a result, two ensembles of the same size, drawn independently from the same training procedure, can differ in their constituent models enough that ensemble prediction lies near a decision boundary for certain test points, making the predicted label unstable. While asymptotically, large ensemble predictions converge to posterior predictive decision and eliminate variability by law of large numbers, practical constraints (eg cost, latency, licensing) often limit ensembles to small sizes, where variance due to finite sampling remains non-negligible. Thus predictive inconsistency across ensembles is an inherent finite sample phenomenon, and motivates the need for instance-specific measures that can identify which instances cannot be trusted to produce stable predictions for fixed-size ensembles.

2

Problem Setup

Notation: For a positive integer n, let [n] = {1, 2, . . . , n}. Random variables will be denoted in boldface, with realizations being non-bolded, so x is a random variable and x is a realization.

2.1

Binary classification

We consider the standard statistical learning setting for binary classification2 . Let X = Rd and Y = {−1, +1} be a space of feature vectors and labels, respectively. Let Dtrain = {(xi , yi ) ∈ X × Y : i ∈ [N ]} denote the training data. Here xi denotes the i-th realization of the feature vector x ∈ X and yi the label corresponding to xi . We assume the pairs in Dtrain are generated by independent and identically distributed (i.i.d.) draws from an unknown distribution π on X × Y. A training algorithm T is a randomized map from X N × Y N → F, where F = {fθ (x) : X → [0, 1] : θ ∈ Θ} is a set of models parameterized by θ ∈ Θ. We interpret the model fθ ∈ F as approximating a probability distribution π̂(y|x) on Y where π̂(y = 1|x) = fθ (x). The function fθ would correspond to the
output of a softmax operation on logit values associated with Y. We define hθ = 1 fθ ≥ 21 as the “hard” decision function associated with fθ . 2 We focus on binary classification to keep our notations simple. Our setting can be extended to multi-class classification by fixing any class and analyzing the predicted probability of that class.

6

We consider the setting where T selects parameters/weights for a fixed DNN architecture by minimizing a loss function, ℓ : Y × Y → R (for example, cross entropy loss), resulting in the population risk L(fθ , x, y) = Ex,y∼π [ℓ(fθ (x), y)].

(1)

The data-generating distribution π is unknown, so we replace the population risk with the empirical risk for training, N

1 X (ℓ(fθ (xi ), yi )). N

L̂(fθ , Dtrain ) =

(2)

i=1

Since the training algorithm itself may be randomized, we can think of it as sampling a function fθ ∈ F. ′ ′ i.i.d. We assume that we are also given a test set Dtest ∈ X N × Y N also sampled ∼ from π. The test error for model fθ with a decision function hθ is N′

1 X [1(hθ (xi ) ̸= yi ))]. err(f ˆ θ) = ′ N

(3)

i=1

For notational simplicity, we henceforth suppress the parameter θ and use f to denote fθ .

2.2

The empirical Rashomon set and predictive multiplicity

We are interested in the Rashomon set of models (Breiman, 2001; Semenova et al., 2022), which captures a class of competing models with similar aggregate predictive performance. Following Hamman et al. (2025), we define the Rashomon set relative to a reference model f0 with satisfactory predictive performance. Definition 1 (Empirical Rashomon set). Given a hypothesis class F , a dataset Dtest = {(xi , yi ), i ∈ [N ′ ]}, a reference model f0 ∈ F , the empirical error err(f ˆ 0 ), defined in Equation 3, and an error tolerance ϵ ∈ (0, 1), the ϵ-empirical Rashomon set is R(ϵ) = {f ∈ F : err(f ˆ ) ≤ err(f ˆ 0 ) + ϵ}.

(4)

Once again, we suppress the dependence on ϵ and write R in place of R(ϵ) to simplify our notations. The Rashomon set R contains models whose empirical error is within ϵ of the reference model f0 . The choice of ϵ determines the range of models considered to be equally well-performing and is application dependent (Marx et al., 2020; Watson-Daniels et al., 2023). Models in the Rashomon set may nevertheless produce different predictions on individual instances. In particular, an instance that is misclassified by one model may be correctly classified by another, while some instances may receive different predictions across many models in R. This phenomenon, known as predictive multiplicity (Marx et al., 2020), can be quantified in several ways. We review the multiplicity metrics considered in this work in Section A.5.

2.3

Ensembles using the Rashomon set

We propose using ensembles of models sampled from the empirical Rashomon set as a proxy for the full Rashomon set. This involves running the training algorithm several times to generate the models. By using rejection sampling, we can assume that T is sampling from a distribution τ on R. Conditioned on Dtrain , repeatedly running the algorithm T generates conditionally i.i.d. samples f1 , f2 , . . . , fM . We define the Rashomon retail as this set of samples R = {fk : k ∈ [M ]} ⊆ R.

7

(5)

As mentioned in Section 1.3, the basic approach to ensembling is to average the constituent model outputs. Given a realization R of the retail set, we define the ensemble of the retail set at x as 1 f¯(R, x) = M

M X

fk (x).

(6)

k=1

By modeling an ensemble of the retail set, f¯(R, x), as randomly sampled from a Rashomon set, we make explicit that ensemble predictions are themselves random variables. Even when all constituent models are near-optimal, different realizations of f¯(R, x) may lead to different ensemble predictions at the same test point. The inconsistency in predictions arising from this randomness is the core difficulty addressed in this paper. In Appendix A.6, we describe other measures that we can compute using a single retail of size M , and in Section 4.3, we show how these measures relate to multiplicity. We also demonstrate that our proposed consistency measure, defined in the next section, performs better at preemptively capturing multiplicity than related measures.

3

Proposed consistency measure

To probe local behavior around a test instance x, we introduce small random perturbations to the input and evaluate the ensemble’s predictions on these perturbed points. For example, we may perturb the raw input x ∈ X by adding noise when looking at deep ensembles. If we are studying ensembles produced by fine-tuning a transformer-based model, we could perturb the embedding, which serves as input to the fine-tuning layers. We can then use the perturbed values to test if the ensemble is consistent at a given input x.

3.1

Measuring ensemble consistency

For simplicity, we will prove our results for additive noise drawn from the uniform distribution on a sphere, although our results should hold for any isotropic subgaussian noise distribution (Vershynin, 2018). For (d) a fixed x, define S2 (x, σ) = {v : ∥x − v∥2 = σ} to be the sphere of radius σ centered at x. (d) Let V = {vj : j ∈ [S]} be drawn i.i.d. from a uniform distribution Unif(S2 (x, σ)). We will measure how sensitive predictions are to small perturbations of x. Definition 2 (Ensembles with perturbed inputs). For a fixed x, Rashomon retail set R, and noise realizations vj , for each j we define the ensemble value at vj by 1 f¯(R, vj ) = M

M X

fk (vj ).

(7)

k=1

Averaging over the S noise realizations, we have 1 f¯(R, V ) = M

M S X 1X k=1

S

fk (vj ).

(8)

j=1

To understand the stability of an ensemble’s predictions at x, we can analyze the ensemble from an independently sampled retail set. The goal is to develop a test based on (x, R, V ) that can identify instances x that are inconsistent. In some applications, these instances will then be reviewed by a more complex system or a human auditor. We would like to characterize test instances x, whose predictions should be stable with respect to any retail sets drawn from the Rashomon set. To that end, we propose a novel consistency measure called the (β, σ)-consistency. We first define consistent test points in Definition 3 as those that are assigned high 8

scores of consistency under the expected model in the full empirical Rashomon set R. We then show that if an individual test point is consistent with respect to the expected model, under some mild assumptions, the ensemble of a retail set R, will also be consistent with high probability (exponential in the number of samples S from a local neighborhood around x and the size M of the retail set). i.i.d.

Definition 3 ((β, σ)-consistency of R). Given R with distribution τ , a random model f ∼ τ , an instance x ∈ Dtest , the expected model Ef [f (x)] of the empirical Rashomon set R at x, a random perturbation v i.i.d.

(d)

∼ Unif(S2 (x, σ)), and parameters β ∈ [− 12 , 12 ] and σ > 0, let cσ (R, x) = Ef [f (x)] −

1 − Ef ,v [|f (x) − f (v)|], 2

(9)

be the margin of the expected model of R at x, penalized by the expected variability around x, where the expectation is taken jointly with respect to the perturbation v, and the model f . Then the expected model of R is said to be (β, σ)-consistent at x if, cσ (R, x) ≥ β.

(10)

The consistency condition in Definition 3 characterizes test instances at which the margin of the empirical Rashomon set’s expected model remains large after accounting for local variability. Our theoretical guarantees require two additional conditions, namely stability of model predictions within the Rashomon set and local Lipschitz continuity. We formalize these conditions below. Definition 4 (ϵ2 -stability of R at x). Given a Rashomon set R, a test point x ∈ Dtest , and a small constant ϵ2 > 0, R is said to be ϵ2 stable at x if, |f (x) − Ef [f (x)]| ≤ ϵ2 , for all f ∈ R.

(11)

Definition 5 (Local Lipschitz continuity). A function f : A ⊂ X → R is said to be locally Lipschitz if for each x ∈ A there exists constant η > 0 and σ > 0 such that, (d)

|f (x) − f (v)| ≤ η∥x − v∥2 , for all v ∈ S2 (x, σ).

(12)

Global Lipschitz continuity requires that a model’s output cannot change too rapidly anywhere in the input space. For modern machine learning models, particularly deep neural networks and large language models, this is both unrealistic and unnecessary. These models are known to exhibit highly non-smooth and complex behavior in regions of the input space far from the data distribution’s support, even when they behave predictably near the data. Imposing a global Lipschitz constraint would therefore exclude most practical models or require overly conservative constants that are dominated by worst-case behaviors irrelevant to the test instances of interest. Rather than assuming smoothness everywhere, we (d) require smoothness only within a neighborhood S2 (x, σ). Definition 6 ((β−ϵ1 , σ)-consistency of a retail set). Given a Rashomon retail set R = {fk : k ∈ [M ]} ⊆ R (d) of size M , a set of perturbed samples V = {vj : j ∈ [S]} ⊆ S2 (x, σ), the ensemble prediction f¯(R, x) at x as in Equation (6), and a constant ϵ1 ≥ 0, let cσ (R, V, x) be defined as, 1 1 cσ (R, V, x) = f¯(R, x) − − 2 S

S M X 1 X j=1

M

fk (x) − fk (vj ) .

(13)

j=1

Then R is said to be (β − ϵ1 , σ)-consistent if cσ (R, V, x) ≥ β − ϵ1 9

(14)

To prove probabilistic guarantees on our proposed consistency measure, we make some common assumptions. Assumption 1 (Rashomon consistency assumptions). The set R = {fk : k ∈ [M ]} is an i.i.d. sample from a distribution τ on R. Every model f in the Rashomon set R is η-locally Lipschitz as in Definition 5. For fixed parameters β, σ and ϵ2 , x belongs to the subset of test instances for which the Rashomon set is (β, σ)-consistent according to Definition 3, and ϵ2 -stable according to Definition 4. We denote this set cons (β, σ, ϵ ) = {x ∈ D as Dtest 2 test : cσ (R, x) ≥ β and R is ϵ2 stable at x} ⊆ Dtest . cons (β, σ, ϵ ). For notational The concentration results that follow are stated only for a fixed x ∈ Dtest 2 cons simplicity, we suppress the dependence of Dtest on β, σ, ϵ2 . Theorem 1 provides a probabilistic guarantee that an ensemble of a randomly drawn retail set R, is (β − ϵ1 , σ)-consistent at x under Assumption 1.

Theorem 1 (Consistency guarantee). Let R, R, and x satisfy Assumption 1 for parameters β ∈ [− 12 , 12 ], cons , let V = {v : j ∈ [S]} be a random set of perturbations of x, and σ, ϵ5 > 0. For a test instance x ∈ Dtest j i.i.d. (d) ¯ distributed ∼ Unif(S2 (x, σ)), and let f (R, x) be the ensemble prediction at x as in Equation (6). Given P 1 PM the consistency measure cσ (R, V, x) = f¯(R, x) − 12 − S1 Sj=1 M k=1 fk (x) − fk (vj ) from Definition 6, for all ϵ1 ≥ 0,       −M ϵ21 −M ϵ21 −Sϵ21 − 2 exp . PR,V [cσ (R, V, x) ≥ β − ϵ1 ] ≥ 1 − 2 exp − 2 exp 8η 2 σ 2 8η 2 σ 2 8ϵ22

(15)

Proof. We refer the readers to Section A.3 for the proof. Theorem 1 states that if x is (β, σ)-consistent under the expected model in the Rashomon set R, then with probability increasing exponentially in both M and S, over the random draws of the retail set, x cons , the theorem provides no will remain (β − ϵ1 , σ)-consistent under the resulting retail sets. For x ∈ / Dtest consistency guarantee: the local variability of the ensemble predictions may be too large, the prediction cons does not imply x is may be too close to the decision boundary, or both. Failure to belong to Dtest inconsistent under the resulting retail sets, rather, the theorem is simply inconclusive for such points.

3.2

Interpretation of the consistency measure

Our goal is to determine, for a fixed instance x, whether the prediction produced by a finite ensemble sampled from a Rashomon set is consistent, or whether it is likely to change if a different ensemble were sampled. In Definition 6, we introduce a consistency measure cσ (R, V, x) that achieves this goal. The first term captures how far the ensemble prediction at x lies from the decision boundary, i.e., the margin gap, rather than raw confidence. This distinction is important, as prior consistency measures often rely either on hard-label agreement (Black et al., 2022b), which discards useful soft information, or on a single model’s confidence, which can be misleading in the presence of multiplicity. More recent approaches (Hamman et al., 2025, 2023) incorporate confidence penalized by variance for a specific class, but can assign low scores to consistent predictors (see Section 4.3.1). In contrast, our margin-based formulation avoids this issue by evaluating distance from the decision boundary, ensuring that such predictors are correctly identified as consistent. However, margin alone does not guarantee robustness. In a Rashomon set, models may agree at x while having decision boundaries arbitrarily close to it, leading to instability under small perturbations. The second term in cσ (R, V, x) explicitly penalizes this by measuring the average variability of predictions of constituent models in the ensemble between x and nearby perturbed points v. This term acts as a local proxy for predictive multiplicity, capturing whether agreement at x persists in its neighborhood. Crucially, unlike prior work that evaluates consistency with a single competing model, our approach leverages ensembles of models sampled from the Rashomon set to reduce variance. Empirically, we 10

show that combining this consistency measure with ensembling reduces predictive multiplicity, mitigates variability in audit decisions, and lowers the risk of incorrect predictions going unchecked. Theorem 1 shows that under Assumption 1, retail ensembles of size M increasingly recover test cons (β, σ, ϵ ) as (β − ϵ , σ)-consistent with an increase in the number of perturbations S and instances in Dtest 2 1 the size M of the retail set. The purpose of the theorem is to establish the concentration phenomenon of retail ensembles’ consistency score rather than to prescribe the choice of ϵ1 or β. Since the proof relies on the distribution-free McDiarmid and Hoeffding inequalities, the resulting finite sample bound is conservative. Accordingly, we fix ϵ1 to a small positive value in our experiments and defer a discussion on the empirical selection of β to Section 4.2.2. To empirically validate this concentration phenomenon, we repeatedly sample retail ensembles of size M from the Rashomon set followingPthe experimental protocol described in Section 4.1. For each retail 1 ensemble, we compute |Dcons (β,σ,ϵ cons (β,σ,ϵ ) 1(cσ (R, V, x) ≥ β − ϵ1 ), the fraction of test instances x∈Dtest 2 2 )| test cons in Dtest (β, σ, ϵ2 ) identified as (β − ϵ1 , σ)-consistent by the retail ensemble. Figure 1 shows the empirical distribution of this quantity over 500 randomly sampled retail ensembles for different values of M . Even for small-sized retail ensembles, the fraction is close to one, indicating that most retail ensembles recover nearly all (β, σ)-consistent test instances as (β−ϵ1 , σ)-consistent. As M increases, the distributions become more concentrated near one, demonstrating increasingly consistent recovery across random ensemble draws, in agreement with Theorem 1.

(a) Binary classification on the Bank dataset using ensembles of fine-tuned BIG-SCIENCE T0 models.

(b) Paraphrase detection on the MRPC dataset using ensembles of fine-tuned BERT models.

Figure 1: Empirical illustration of the concentration guarantee in Theorem 1. For each of 500 indepencons dently sampled retail ensembles of size M , we compute the fraction P of test instances in Dtest (β, σ, ϵ2 ) that 1 remain (β − ϵ1 , σ)-consistent under the retail set, |Dcons (β,σ,ϵ2 )| x∈Dcons (β,σ,ϵ2 ) 1(cσ (R, V, x) ≥ β − ϵ1 ). As test test the retail ensemble size increases, the distributions become increasingly concentrated near 1, indicating that most randomly sampled retail ensembles recover nearly all (β, σ)-consistent test instances as (β − ϵ1 , σ)-consistent.

4

Experimental Setup

The theoretical results in Section 3 establish that the proposed consistency measure identifies test instances whose predictions remain consistent across finite ensembles sampled from the Rashomon set. In this section, we empirically evaluate whether these guarantees translate into practical benefits for auditing

11

and predictive multiplicity analysis. Our experiments are designed to answer three questions that directly follow from our theory. • Can the proposed consistency measure be used as an effective auditing criterion? Consistency in predictions alone does not imply correctness in predictions. A prediction may be stable across models and still be incorrect. Our objective is not to show that consistency perfectly identifies incorrect predictions, but rather to investigate the measure’s usefulness in auditing. We therefore evaluate whether thresholding the proposed consistency score produces a favorable trade-off between audit risk (missed erroneous predictions) and audit cost (unnecessary diversions of correct predictions). We do not attempt to optimize the auditing policy itself or compare against every possible rejection strategy, since our goal is to evaluate the usefulness of the consistency measure rather than design a novel auditing framework. We find that increasing the consistency threshold can reduce audit risk, in some setups close to 0, while requiring a only moderate increase in diversions. • Do finite retail ensembles approximate the audit decision of the Rashomon set as ensemble size increases? Theorem 1 shows that consistency estimates obtained from finite ensembles should concentrate around those of the expected model of the Rashomon set. We therefore study how diversion decisions and audit outcomes change as the ensemble size M increases. We observe that as the number of models in the retail ensemble increases, auditing outcomes become increasingly concentrated around the ensemble model of the full Rashomon set. This empirical behavior is consistent with the theoretical guarantees established in Section 3. Since enumerating the Rashomon set is infeasible, practitioners can only deploy ensembles with a finite number of models due to constraints on computation, latency, or memory. Demonstrating that relatively small retail ensembles approximate the Rashomon set behavior is therefore essential for showing that the proposed method is practically deployable. We intentionally restrict our study to the standard ensemble formed by uniformly averaging model predictions, since this estimator is the one analyzed throughout our theoretical development. Alternative ensemble strategies, including learned weighted ensembles and mixture-of-experts architectures, assign non-uniform or input-dependent weights to constituent models, thereby changing the estimator and requiring separate theoretical analysis. Understanding how the proposed consistency measure interacts with these more sophisticated ensemble constructions is an interesting direction for future work. • Does the proposed consistency measure capture predictive multiplicity? Accurate estimation of existing multiplicity metrics outlined in Section A.5 requires access to the Rashomon set and is therefore difficult to estimate in practice. We investigate whether the proposed consistency measure, which jointly accounts for ensemble margin and local prediction stability, provides a more informative characterization of predictive multiplicity than prior measures that consider individual models or rely solely on confidence scores. Our experiments show that our proposed consistency measure exhibits strong correlations with established multiplicity metrics. By combining ensemble margin and local perturbation stability, our measure also shows stronger correlations with multiplicity metrics than other related measures while remaining practical to estimate from finite retail ensembles. The remainder of this section is organized as follows. In Section 4.1, we describe the experimental protocol and how we construct the Rashomon set and sample retail ensembles from it. In Section 4.2, we answer Questions 1 and 2, and in Section 4.3, we answer Question 3.

4.1

Scenario Specification

To avoid potential application-specific bias, we implement the proposed consistency measure in two disparate application domains, natural language processing (NLP) and classification of tabular data, and 12

evaluate the behavior of the consistency measure with respect to ensembles drawn from the respective Rashomon sets. In the NLP domain, we consider two binary classification tasks from the GLUE benchmark. The first is sentiment classification, using the SST-2 dataset, (Socher et al., 2013). SST-2 consists of movie reviews labeled as positive or negative. We use approximately 67,000 instances for training and 872 instances for validation. The second task is paraphrase detection, using the MRPC dataset (Dolan & Brockett, 2005). MRPC contains sentence pairs labeled as semantically equivalent, with approximately 3,700 training instances and 408 validation instances. For both tasks, we fine-tune encoder-only transformer architectures, namely BERT (Devlin et al., 2019) and RoBERTa (Zhuang et al., 2021), using standard supervised fine-tuning. In the second evaluation setting, we consider binary classification on tabular data using the Adult (Becker & Kohavi, 1996) and Bank Marketing (Moro et al., 2014) datasets. The Adult dataset predicts whether an individual’s annual income exceeds $50, 000 based on demographic and employment-related attributes, while the Bank Marketing dataset predicts whether a client subscribes to a term deposit using demographic, financial, and campaign-related features. We adopt the T-Few prompt-based fine-tuning framework (Liu et al., 2022), which reformulates each row of a tabular dataset as natural language before it is processed by a large language model. Following this framework, each tabular instance is serialized by representing each feature along with its corresponding value, and then combined with a task-specific prompt. This serialization enables structured tabular prediction to be formulated as a natural language understanding problem. We use the BIG-SCIENCE T0 encoder-decoder language model (Sanh et al., 2022) together with IA3 adapters (Liu et al., 2022) for parameter-efficient fine-tuning. We adopt this experimental protocol because it uses the same T-Few framework as Hamman et al. (2025), enabling comparison with their analysis of predictive multiplicity under fine-tuning. The source of randomness depends on the fine-tuning strategy. For encoder-based models (BERT and RoBERTa) on NLP tasks, all runs start from the same pre-trained backbone, while the random seed determines the initialization of the task-specific classification head and other stochastic components of training, including data shuffling and dropout. For the BIG-SCIENCE T0 models on the tabular tasks, the pre-trained backbone remains frozen, and only the IA3 adapter parameters are randomly initialized and optimized during fine-tuning. 4.1.1

Empirical Rashomon set generation

As previously discussed, it is infeasible to enumerate the entire empirical Rashomon set for any of the scenarios we wish to evaluate. Thus, for each dataset-model pair, we approximate the empirical Rashomon set by fine-tuning a large pool of models from the same pre-trained initialization under different random seeds. For each dataset-model pair, we select the model with the lowest empirical error in the pool and choose ϵ = 0.03 to define the Rashomon set relative to its empirical error following Definition 1. We then apply rejection sampling to retain T = 100 models that satisfy the Rashomon set criterion, yielding our empirical approximation of the Rashomon set 3 . The resulting set of T retained models is referred to as the Rashomon warehouse and is denoted by RT . 4.1.2

Selecting the radius σ of local perturbations

For both the encoder-based models and the T-Few framework, we compute the proposed consistency measure in the model’s representation space. Each input is first mapped to an embedding representation before being processed by the fine-tuning layers. We abuse notation and use x ∈ X to denote both the original input and its embedding representation. Thus, all perturbations described in this section are 3

The authors acknowledge the Office of Advanced Research Computing (OARC) at Rutgers, The State University of New Jersey, for providing access to the Amarel cluster and associated research computing resources that have contributed to the results reported here (University, 2026). The choice of T as a result is influenced by the available GPU resources and per-job walltime limits on the cluster

13

applied to x in the embedding space. Following Hamman et al. (2025), we set the perturbation radius as σ = α · medianx (∥x − NN(x)∥2 ),

(16)

where NN(x) denotes the nearest neighbor of x in the training set and α is a scaling parameter. The median nearest neighbor distance provides a data-driven estimate of the size of the local neighborhood around a sample. Choosing σ as a fraction of this distance ensures that perturbations remain within the local neighborhood of each test instance, allowing the proposed consistency measure to evaluate local prediction stability rather than probing behavior under unrealistically large perturbations. A radius of σ = 0.01 computed this way worked well across all settings. Similar neighborhood-based scaling is commonly used in machine learning algorithms, where hyperparameters are chosen relative to the intrinsic geometry of the data, such as in the literature of certified robustness techniques (Cohen et al., 2019; Salman et al., 2019; Alfarra et al., 2022). 4.1.3

Computing retail set consistency

From the Rashomon warehouse RT , we sample M models, to form a retail set RM , of size M , to reflect realistic deployment scenarios where only a limited ensemble size is feasible. We explore different values of M , i.e., M = [1, 3, 5, 10, 30], to analyze the benefits of ensembling more models over individual models. We estimate consistency using Monte Carlo samples drawn uniformly from the surface of the l2 sphere of radius σ centered at x. For each consistency computation, all models in a sampled retail ensemble must be evaluated on the same perturbation set. Ideally, a new perturbation set would be generated for every Monte Carlo sample of the retail set, as well as for different sizes M of the retail set. However, this would require rerunning model inference for every replication and every M , which is computationally prohibitive, since we observed GPU memory limits restrict inference to at most 100 perturbations per forward pass. Instead, we use T independently trained models in the warehouse to predict on a library of P = 200 randomly generated perturbations for each x. Each model is evaluated once for each perturbation in the perturbation library, and the resulting predictions are cached. For each Monte Carlo replication q = 1, . . . ., Q, we sample without replacement the ensemble predictions corresponding to a retail set RM,q of size M , and perturbation set VS,q of size S to compute the consistency score cσ (RM,q , VS,q , x) for the q − th draw. Repeating this procedure for Q = 500 replications yields an empirical distribution of the consistency estimator that captures the variability due to both retail ensemble sampling and perturbation sampling. For comparison, the Rashomon warehouse consistency score is computed once using the entire warehouse ensemble RT together with the perturbation set VP , yielding cσ (RT , x). Code to reproduce the experiments in this paper will be made available shortly, following final cleanup4 .

4.2

Operationalizing the consistency measure through auditing

We return to the auditing framework introduced in Section 1.1, where we compared the decisions made using an idealized baseline reference model and a practical candidate model. Table 1 enumerated all possible combinations of correct/incorrect predictions and diversion decisions of these models. The remaining question is how to decide when to divert an input. Our proposed approach decides when to divert by thresholding the consistency measure. The complete decision procedure is summarized in Algorithm 1. Given a set of models R, let f¯(R,
x) denote the ensemble prediction for input x, which produces an estimated label h(x; R) = 1 f¯(R, x) ≥ 12 and an error e(x; R) = 1(h(x; R) ̸= y). Specifically, we define the estimated model risk as r(x; R) = 1(cσ (R, V, x) ≥ β − ϵ1 ), where β − ϵ1 is the audit threshold for retail set ensembles. Inputs satisfying cσ (R, V, x) ≥ β − ϵ1 are diverted for human review, while the ensemble prediction is accepted for the remaining inputs. 4

https://github.com/Sinjini77/Measuring-ensemble-consistency

14

In the subsequent sections, we show how varying β induces a trade-off between audit cost and audit risk, and how this trade-off can guide the choice of β. We characterize this trade-off using the Rashomon warehouse ensemble, which serves as our baseline model and represents the reference auditing strategy against which finite sized ensembles are compared. Decreasing β reduces the number of audited instances, thereby decreasing unnecessary human reviews of correctly classified samples, but may also increase the number of incorrect predictions that bypass auditing. Conversely, increasing β results in more conservative auditing, reducing missed errors at the expense of auditing more correct predictions. Finally, we evaluate whether the auditing decisions made using retail ensembles of fixed size M approach those of the Rashomon warehouse ensemble as the ensemble size increases. Consistent with the objective outlined in Section 1.1, a desirable auditing strategy is one that minimizes excess audit cost and risk by reproducing the warehouse baseline’s audit outcomes while requiring substantially fewer models. Algorithm 1 Retail set ensembling consistency test Input: Test point x, Rashomon retail set R = {fk : k ∈ [M ]} of M models, threshold β, slack parameter ϵ1 Output: Decision to divert x to an expert i.i.d. (d) Sample S points V = {vj : j ∈ [S]} ∼ Unif(S2 (x, σ)). Compute cσ (R, V, x) as in Definition 6. Test if cσ (R, V, x) ≥ β − ϵ1 if Test is false then Ensemble diverts else Ensemble predicts end if

4.2.1

Audit metrics

We quantify the trade-off between risk and cost highlighted in the previous section using the auditing metrics introduced below, which measure both unnecessary audits for correct predictions and erroneous predictions that escape human review. Definition 7 (Costs and risks for auditing). Let R be a retail set and f¯(R, x) be the ensemble prediction of R for a fixed instance x ∈ Dtest , with N ′ as the size of Dtest . Let h(x; R) be the model’s estimated label, e(x; R) = 1(h(x; R) ̸= y) the error label, and r(x; R) = 1(cσ (R, V, x) ≥ β − ϵ1 ) be the risk assessment. The costs and risks of the auditing system in Algorithm 1 can be measured in several ways. • Relative cost and risk. The relative cost and risk compare the auditing system behavior in Algorithm 1 in relation to the ensemble model predictions. Define the following two ratios: P 1(e(x; R) = 0, r(x; R) = 1) P ratiocost (R) = x∈Dtest (17) x∈Dtest 1(e(x; R) = 0) P 1(e(x; R) = 1, r(x; R) = 0) P ratiorisk (R) = x∈Dtest (18) x∈Dtest 1(e(x; R) = 1) The audit cost ratio in (17) is the fraction of points that the ensemble classified correctly but the auditing system sent for human intervention. The audit risk ratio in (18) is the fraction of points that the ensemble classified incorrectly, but the auditing system did not send for human intervention. • Absolute cost and risk. We also measure the overall auditing burden and audit risk as fractions

15

of the entire test set. Define the following two fractions: P 1(r(x; R) = 1) fracdiv (R) = x∈Dtest N′ P 1(e(x; R) = 1, r(x; R) = 0) fracrisk (R) = x∈Dtest N′

(19) (20)

The diversion fraction (19) is the fraction of the test set sent for human intervention. The audit risk fraction (20) is the fraction of the test set on which the auditing system fails to divert an erroneous prediction. We use ratiorisk (RM ), ratiocost (RM ), fracdiv (RM ), and fracrisk (RM ) to analyze audit behavior of a retail set RM , of size M . Similarly, for a Rashomon warehouse RT , with ensemble prediction f¯(RT , x), estimated label h(x; RT ), error label e(x; RT ) = 1(h(x; RT ) ̸= y), and risk assessment r(x; RT ) = 1(cσ (RT , x) ≥ β), we use ratiorisk (RT ), ratiocost (RT ), fracdiv (RT ), and fracrisk (RT ) to analyze audit behavior of the Rashomon warehouse. 4.2.2

High consistency thresholds substantially reduce audit risk for Rashomon warehouse

In this section, we study how the auditing behavior of the Rashomon warehouse ensemble varies with the audit threshold β in Equation (10). The threshold determines the audit decision r(x; RT ) = 1(cσ (RT , x) ≥ β) and therefore controls the trade-off between audit risk and audit cost in Definition 7. Throughout this section, we use paraphrase detection on the MRPC dataset with an ensemble of fine-tuned BERT models as a representative example. Results for additional datasets and architectures are provided in Appendix A.7.1. Figure 2a reports the warehouse audit cost ratio, ratiocost (RT ), and audit risk ratio, ratiorisk (RT ) for thresholds β = [−0.40, −0.30, −0.20, −0.10, 0.0, 0.10, 0.20, 0.25, 0.30, 0.35, 0.40, 0.45, 0.50], using a coarser grid for smaller values of β and finer grid for β ≥ 0.2. Since our goal is to identify highly consistent test instances, the region of larger β values is of greater practical interest, and the finer resolution allows us to more accurately characterize the corresponding trade-off between audit cost and audit risk. As β increases, the auditing system becomes more conservative, diverting more test instances for human review. As a result, the audit risk ratio decreases while the audit cost ratio increases. Importantly, as we increase β, the audit risk ratio falls much more rapidly than the audit cost ratio rises, demonstrating the benefits of setting β to higher values. This behavior is particularly desirable in high-risk applications, where reducing missed errors is more valuable than avoiding a moderate increase in human review. Figure 2b reports the warehouse diversion fraction and audit risk fraction as functions of β. Increasing β substantially reduces the fraction of test instances that incur audit risk before causing a sharp increase in the diversion fraction. For example, increasing β from −0.5 to 0.25 reduces the audit risk fraction (represented by the solid red line) from approximately 0.174 to almost 0.0. Based on this, we set β using a held-out test set to evaluate the trade-off between diversion fraction and audit risk fraction. We explicitly specify the maximum diversion fraction we are willing to incur and set an upper bound equal to the ensemble’s prediction error plus an additional tolerance of 0.15. Starting from β = −0.5, we increase β in increments mentioned above and compute the resulting diversion fraction. We then select the largest value of β for which the diversion fraction does not exceed the prescribed upper bound. This procedure allows us to tolerate only a modest increase in diversions beyond the baseline prediction error while maximizing the reduction in audit risk. Across all datasets and models considered in this work, this criterion consistently identifies operating points with audit risk fraction close to 0.05 without incurring a substantial increase in diversion fraction. 4.2.3

Retail set audit outcomes match warehouse audit outcomes as we increase M

Section 4.2.2 showed that thresholding consistency scores provides an effective auditing criterion for the Rashomon warehouse. In practice, however, consistency scores must be estimated from a finite retail 16

(a) Comparing audit cost ratio (ratiocost (RT )) and risk ratio (ratiorisk (RT )). Increasing the threshold β decreases risk at the expense of a higher cost.

(b) Comparing diversion fraction (fracdiv (RT )) and audit risk fraction (fracrisk (RT )). Increasing the threshold β causes the fraction of diverted points to increase and brings down the overall risk close to 0.

Figure 2: Comparing the performance metrics in Definition 7 for the warehouse ensemble model as a function of the threshold β used in the consistency check of Algorithm 1. Results are shown for paraphrase detection on the MRPC dataset using an ensemble of fine-tunings of the BERT model. ensemble. In this section, we investigate how closely auditing decisions based on retail ensembles approximate those of the warehouse as the ensemble size M increases. We fix the audit threshold based on our discussion in Section 4.2.2, and evaluate retail ensembles of size M = [1, 3, 5, 10, 30]. As described in Section 4.1.1, we generate Q = 500 independent retail ensembles RM,q and compute the audit metrics ratiocost (RM,q ), ratiorisk (RM,q ), fracdiv (RM,q ), and fracrisk (RM,q ) from Definition 7. Figures 3a and 3b report the mean ± standard deviation over the Q draws for paraphrase detection on the MRPC dataset using ensembles of independently fine-tuned BERT models. Results with additional datasets and architectures are provided in Appendix A.7.2. The concentration result in Theorem 1 implies that retail consistency scores converge to the warehouse consistency scores as M and S increase. Consequently, retail auditing decisions based on the consistency scores should increasingly align with those of the warehouse ensemble. Rather than using the potentially conservative theoretical value of ϵ1 , we adopt a practical tolerance of ϵ1 = 0.01, yielding a retail auditing threshold of β − ϵ1 . Figure 3a shows that small retail ensembles exhibit substantially higher audit risk ratios than the warehouse ensemble. For no ensembling, and ensemble size M = 3, the mean audit risk ratio (represented by the solid yellow line) is approximately 0.4 and 0.25, respectively. In contrast, the corresponding warehouse value (represented by the dashed yellow line) is much lower, around 0.15. As M increases, the audit risk ratio steadily decreases toward the warehouse value, while the audit cost ratio increases toward the warehouse audit cost ratio, indicating that larger retail ensembles more faithfully reproduce the warehouse auditing decisions. A similar trend is observed in Figure 3b. The audit risk fraction corresponding to no ensembling (M = 1) represented by the solid red line, is close to 0.07. In contrast, for the warehouse ensemble represented by the dashed red line, the value is close to 0. The audit risk fraction decreases rapidly as M increases, while the diversion fraction decreases to align with the warehouse diversion fraction. Most of the improvement occurs between M = 3 and M = 10, after which the curves largely stabilize. Overall, these results provide empirical support for Theorem 1. As the retail ensemble size increases, retail consistency scores and the resulting audit decisions converge toward those of the warehouse ensemble. Importantly, this behavior is achieved with relatively modest ensemble sizes, suggesting that practical

17

retail ensembles can closely approximate the auditing performance of the Rashomon warehouse without requiring an excessive number of models. A more detailed analysis of the individual audit outcomes, including how retail and warehouse audit decisions differ across the outcome categories introduced in Table 1, is provided in Appendix A.7.3.

(a) Comparing the audit cost ratio (ratiocost ) and risk ratio (ratiorisk ) of retail sets for different ensemble sizes. Increasing the ensemble size has a small impact in matching warehouse once the ensemble is sufficiently large (around M= 10).

(b) Comparing the diversion fraction (fracdiv ) and audit risk fraction (fracrisk ) of retail sets for different ensemble sizes. Diversion and risk fractions nearly match those of the warehouse, even with relatively modest ensemble sizes.

Figure 3: Comparing the audit metrics in Definition 7 for retail ensemble models as a function of the number of models M . For each size M , 500 ensembles of size M were generated by sampling with replacement from the warehouse set. Figures show the mean ± standard deviation (std) of the metrics in Definition 7 taken over the 500 retail ensembles. Results are shown for paraphrase detection on the MRPC dataset using an ensemble of fine-tunings of the BERT model.

4.3

Comparison of proposed consistency measure against warehouse multiplicity metrics

In the previous sections, we showed that the proposed consistency measure can be used effectively as an auditing criterion. An equally important question is whether it can also pre-emptively capture predictive multiplicity. In this section, we evaluate the extent to which the proposed consistency score computed using a single retail ensemble of size M correlates with multiplicity metrics computed using the Rashomon warehouse. A strong correlation would indicate that consistency computed from a single retail ensemble can serve as a practical proxy for predictive multiplicity in deployment settings where the full Rashomon warehouse is unavailable. For a fixed test instance x ∈ Dtest , the multiplicity metrics in Appendix A.5, pairwise disagreement (Black et al., 2022a), discrepancy (Marx et al., 2020), local discrepancy, prediction variance (Hamman et al., 2025; Watson-Daniels et al., 2023), and prediction range (Hamman et al., 2025; Watson-Daniels et al., 2023) (Section A.5), quantify different forms of disagreement among models in the Rashomon set. Since the Rashomon set is approximated by a warehouse RT of size T = 100, these multiplicity metrics are likewise estimated using this set. For notational simplicity, we suppress the explicit dependence of our proposed consistency measure cσ (x) and other related measures defined in Appendix A.6 on RM and VS . To assess whether the proposed 18

consistency measure captures predictive multiplicity, we analyze the correlation of cσ (x) with these warehouse multiplicity metrics. We repeat this analysis for several related consistency measures that can also be computed from a single retail ensemble, to compare them against our proposed consistency measure. In particular, we compare against the local ensemble consistency measure, lmcσ (x) (Definition 10), for which we also establish concentration guarantees in Appendix A.4.1. While our proposed consistency measure evaluates the ensemble margin at the test instance and penalizes it by the average local prediction variability, lmcσ (x) instead computes the average ensemble prediction over the local neighborhood of x before applying the same variability penalty term. We also compare against measures that isolate individual components of consistency, namely margin (mg(x), Definition 17), local margin (lmgσ (x), Definition 18), and local variability (lvσ (x), Definition 19). Finally, we compare with the consistency score hcσ (x) (Definition 20), a prior consistency measure proposed by Hamman et al. (2025) that evaluates the confidence of a single model at the test instance and penalizes it by local prediction variability. As a result, it neither measures the margin relative to the decision boundary nor exploits ensembling over multiple models. For fair evaluation, we compare the measures for different ensemble sizes. By comparing their correlations with the warehouse multiplicity metrics, we evaluate whether our proposed consistency measure provides a more informative characterization of predictive multiplicity. To quantify the relationship between consistency measures and predictive multiplicity, we use the Spearman rank correlation coefficient. Unlike linear correlation, Spearman correlation measures the strength of the monotonic relationship between two quantities based on their rankings, making it well suited for comparing consistency measures and multiplicity metrics that may not exhibit a linear relationship. For each consistency measure and multiplicity metric pair, we compute the Spearman correlation coefficient across the test set, defined as follows. Definition 8 (Spearman rank correlation coefficient). Let a = (a1 , . . . , aN ) and, b = (b1 , . . . , bN )] be two sets of paired observations. Let rank(ai ) and rank(bi ) denote the ranks of ai and bi , respectively, and define di = rank(ai ) − rank(bi ). Assuming there are no tied ranks, the Spearman correlation coefficient between a and b is P 2 6 N i=1 di ρ(a, b) = 1 − . (21) N (N 2 − 1) Table 2 reports these correlation values for retail ensembles of size M = [1, 5, 10]. Rows highlighted in light blue correspond to the proposed consistency measures, cσ (x) and lmcσ (x), while values in bold represent the highest value achieved for a fixed multiplicity metric and a fixed ensemble size. Across all multiplicity metrics and for all ensemble sizes, our proposed consistency measure, cσ (x), and lmcσ (x) achieve the highest correlation among all consistency measures. In particular, our measure exhibits strong correlations with pairwise disagreement, discrepancy, and local discrepancy, and particularly high correlations with prediction variance and prediction range. Test instances with high consistency scores correspond to a large ensemble margin and low variability of model predictions under local perturbations. A large margin indicates that the ensemble prediction lies far from the decision boundary, while low local variability indicates that the prediction is stable in the neighborhood of the test instance. As a result, these instances tend to exhibit low predictive multiplicity among the models in the warehouse, whereas instances with low consistency scores tend to exhibit substantially higher predictive multiplicity. Another important observation is the benefit of computing consistency using a retail ensemble rather than a single model. Across nearly all consistency measures, the correlations with the warehouse multiplicity metrics increase as the retail ensemble size increases from M = 1 to M = 5 and M = 10. This demonstrates that aggregating predictions from independently trained models provides a substantially more informative estimate of predictive multiplicity than a single model can. The gains are particularly pronounced when increasing the retail ensemble size from one to five models, with more modest improvements thereafter, suggesting that relatively small retail ensembles are sufficient to capture much of the multiplicity information present in the Rashomon warehouse. Additional experiments with other architectures and datasets are included in Appendix A.7.5. 19

Figure 4: Scatter plots showing the relationship between consistency scores computed from a retail ensemble (x-axis) and the pairwise disagreement multiplicity metric (Black et al., 2022a), computed using the Rashomon warehouse (y-axis) for individual test instances. The top row corresponds to the proposed consistency measure cσ (Definition 6), while the bottom row corresponds to the consistency score hcσ of Hamman et al. (2025) (Definition 20). Columns show results for different retail ensemble sizes, and each colored dot represents one of 200 randomly sampled test instances. The proposed consistency measure exhibits a substantially stronger correlation with pairwise disagreement across all retail ensemble sizes than hcσ . Results are shown for paraphrase detection on the MRPC dataset using ensembles of independently fine-tuned BERT models. We also note that measures based solely on margin information or solely on local variability perform consistently worse than measures that combine both. While margin captures distance relative to the decision boundary, it does not account for the stability of predictions under local perturbations. As a result, high-confidence predictions can still exhibit substantial multiplicity locally. Conversely, local variability measures only the stability of predictions under perturbations and does not account for how close the ensemble prediction lies to the decision boundary. As a result, predictions with low local variability may still have small margins and therefore be susceptible to predictive multiplicity. Combining both margin and local variability provides a more complete characterization of prediction consistency, which explains the stronger correlations achieved by the rows in blue. 4.3.1

A note on hcσ (x)

The score hcσ (x) in Hamman et al. (2025) exhibits moderate correlations with the multiplicity metrics on MRPC, but its performance deteriorates substantially on SST2, where the correlations are considerably lower than those achieved by the proposed measures. However, the discrepancy between the high correlation of hcσ (x) with the multiplicity metrics in one experiment and the low correlation in the other is not surprising. By definition, hcσ (x) measures the confidence assigned to a class and penalizes it by the average variability of that confidence under local perturbations. As a result, a predictor that consistently predicts the negative class with only moderate confidence can receive a low consistency score despite exhibiting highly stable predictions. An example of this behavior can be seen in Figure 4, where test instances with low multiplicity are assigned both high and low scores by hcσ (x). Several test instances with pairwise disagreement close to 0 are assigned hcσ (x) scores near the extremes, close to 0 or close to 1. Thus, hcσ (x) incorrectly identifies many highly consistent test instances as inconsistent, because it depends on the confidence assigned to the predicted class rather than the distance from the decision 20

boundary. We observe this behavior across all multiplicity metrics, the results of which are included in Section A.7.4 of the Appendix. In contrast, the proposed consistency measure assigns high scores to nearly all test instances exhibiting low pairwise disagreement in the Rashomon warehouse. Instances with pairwise disagreement close to 0 are assigned the highest scores (above 0.4). The low multiplicity of instances with high scores from the proposed consistency measure is by construction, because the score is high for instances with large ensemble margins and low local prediction variability, which implies that predictions across the Rashomon warehouse are stable at these instances. We also note that in Hamman et al. (2025), hcσ (x) is evaluated with respect to the predicted class. As different models sampled from a Rashomon set may predict different classes for the same test instance, the quantity being evaluated changes whenever the predicted label changes. From our perspective, this complicates both the empirical interpretation and theoretical analysis of the measure, since the consistency score no longer corresponds to a fixed underlying quantity across different models. In contrast, our proposed consistency measure is defined with respect to an ensemble prediction for a fixed class, yielding a quantity whose concentration toward the corresponding Rashomon set value can be analyzed theoretically.

5

Conclusion

This work establishes a principled framework for analyzing predictive multiplicity and its impact on downstream auditing using finite ensembles drawn from the Rashomon set. We introduced a consistency measure that combines ensemble margin with local prediction variability, enabling the identification of predictions that remain stable across competing models. We further established theoretical guarantees showing that, under certain mild assumptions, consistency estimates computed from finite ensembles converge to those of the full Rashomon set as the ensemble size and the number of perturbation samples increase. Our experiments demonstrate three key findings. First, finite ensembles provide a trade-off between audit cost and audit risk. Although a single model is considerably less expensive to deploy than approximating the full Rashomon set with larger ensembles, they exhibit much greater variability in auditing decisions. Additionally, individual models result in a higher volume of erroneous predictions escaping human review, but fewer unnecessary diversions, compared to an ensemble of the full Rashomon set. Even with a modest increase in ensemble size, audit decisions become increasingly stable and closely approximate those of the full Rashomon set. Second, contrary to prior work, our results show that using a single model to characterize predictive multiplicity is insufficient. Even relatively small ensembles consistently achieve substantially stronger correlations with established predictive multiplicity metrics, demonstrating that aggregating independently trained models provides a more faithful characterization of multiplicity. Finally, we show that how consistency is measured is equally important. A margin-based consistency measure penalized by local prediction variability consistently outperforms confidence-based approaches across most experimental settings. Several directions remain for future work. Our theoretical analysis considers uniformly weighted ensembles, whereas modern prediction systems increasingly employ weighted ensembles and mixture-ofexperts architectures, indicating that extending the concentration guarantees to these settings would be valuable. While this work focuses on probabilistic classification, extending the proposed consistency measure and auditing framework to regression and generative foundation models presents an important direction for future research. More broadly, we hope this work encourages the study of predictive multiplicity not only as a property of machine learning models but also of their effects on downstream decision systems that rely on them.

21

References Motasem Alfarra, Adel Bibi, Philip H. S. Torr, and Bernard Ghanem. Data-dependent randomized smoothing. In Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence, volume 180, pp. 64–74. PMLR, 2022. URL https://proceedings.mlr.press/v180/alfarra22a. html. Amazon Web Services. AWS Marketplace: Machine learning products, 2026. URL https://aws.amazon. com/marketplace/solutions/machine-learning. Sinjini Banerjee, Tim Marrinan, Reilly Cannon, Tony Chiang, and Anand D. Sarwate. Measuring training variability from stochastic optimization using robust nonparametric testing. IEEE Journal of Selected Topics in Signal Processing, 19(7):1477–1492, 2025. doi: 10.1109/JSTSP.2025.3583140. Peter L. Bartlett and Marten H. Wegkamp. Classification with a reject option using a hinge loss. Journal of Machine Learning Research, 9(59):1823–1840, 2008. URL https://www.jmlr.org/papers/v9/ bartlett08a.html. Barry Becker and Ronny Kohavi. https://doi.org/10.24432/C5XW20.

Adult.

UCI Machine Learning Repository, 1996.

DOI:

Emily Black, Manish Raghavan, and Solon Barocas. Model multiplicity: Opportunities, concerns, and solutions. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pp. 850–863, 2022a. Emily Black, Zifan Wang, Matt Fredrikson, and Anupam Datta. Consistent counterfactuals for deep models. In International Conference on Learning Representations, 2022b. URL https://openreview. net/forum?id=St6eyiTEHnG. Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Nazanin Mohammadi Sepahvand, Edward Raff, Kanika Madan, Vikram Voleti, Samira Ebrahimi Kahou, Vincent Michalski, Tal Arbel, Chris Pal, Gael Varoquaux, and Pascal Vincent. Accounting for variance in machine learning benchmarks. Proceedings of Machine Learning and Systems, 3, 2021. Leo Breiman. Statistical modeling: The two cultures (with comments and a rejoinder by the author). Statistical Science, 16(3):199–231, 2001. Mustafa Cavus and Przemyslaw Biecek. Investigating the impact of balancing, filtering, and complexity on predictive multiplicity: A data-centric perspective. Information Fusion, 123:103243, 2025. ISSN 1566-2535. doi: https://doi.org/10.1016/j.inffus.2025.103243. URL https://www.sciencedirect. com/science/article/pii/S1566253525003161. Martino Ciaperoni, Han Xiao, and Aristides Gionis. Efficient exploration of the Rashomon set of ruleset models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 478–489, 2024. Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In International Conference on Machine Learning, pp. 1310–1320. PMLR, 2019. Gordon Dai, Pavan Ravishankar, Rachel Yuan, Emily Black, and Daniel B Neill. Be intentional about fairness!: Fairness, size, and multiplicity in the Rashomon set. In Proceedings of the 5th ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, pp. 42–73, 2025.

22

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https://aclanthology.org/N19-1423/. Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. Finetuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06305, 2020. William B. Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005. URL https: //aclanthology.org/I05-5002/. Leo Feng, Mohamed Osama Ahmed, Hossein Hajimirsadeghi, and Amir H. Abdi. Towards better selective classification. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=5gDz_yTcst. Aaron Fisher, Cynthia Rudin, and Francesca Dominici. All models are wrong, but many are useful: Learning a variable’s importance by studying an entire class of prediction models simultaneously. Journal of Machine Learning Research, 20(177):1–81, 2019. Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape perspective. arXiv preprint arXiv:1912.02757, 2019. Mudasir A. Ganaie, Minghui Hu, Ashwani Kumar Malik, Muhammad Tanveer, and Ponnuthurai N. Suganthan. Ensemble deep learning: A review. Engineering Applications of Artificial Intelligence, 115: 105151, 2022. doi: 10.1016/j.engappai.2022.105151. URL https://doi.org/10.1016/j.engappai. 2022.105151. Odd Erik Gundersen, Kevin Coakley, Christine Kirkpatrick, and Yolanda Gil. Sources of irreproducibility in machine learning: A review. arXiv preprint arXiv:2204.07610v2, 2023. Faisal Hamman, Erfaun Noorani, Saumitra Mishra, Daniele Magazzeni, and Sanghamitra Dutta. Robust counterfactual explanations for neural networks with probabilistic guarantees. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pp. 12351–12367. PMLR, 2023. URL https://proceedings.mlr.press/v202/hamman23a.html. Faisal Hamman, Pasan Dissanayake, Saumitra Mishra, Freddy Lecue, and Sanghamitra Dutta. Quantifying prediction consistency under fine-tuning multiplicity in tabular LLMs. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu (eds.), Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 21662–21684. PMLR, 13–19 Jul 2025. URL https://proceedings.mlr.press/v267/hamman25a.html. Jianxing He, Sally L Baxter, Jie Xu, Jiming Xu, Xingtao Zhou, and Kang Zhang. The practical implementation of artificial intelligence technologies in medicine. Nature Medicine, 25(1):30–36, 2019. Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.

23

Kilian Hendrickx, Lorenzo Perini, Dries Van der Plas, Wannes Meert, and Jesse Davis. Machine learning with a reject option: A survey. Machine Learning, 113:3073–3110, 2024. doi: 10.1007/ s10994-024-06534-x. URL https://doi.org/10.1007/s10994-024-06534-x. Radu Herbei and Marten H. Wegkamp. Classification with reject option. The Canadian Journal of Statistics, 34(4):709–721, 2006. doi: 10.1002/cjs.5550340410. Miklós Z. Horváth, Mark Niklas Müller, Marc Fischer, and Martin Vechev. Boosting randomized smoothing with variance reduced classifiers. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=mHu2vIds_-b. Hsiang Hsu and Flavio Calmon. Rashomon capacity: A metric for predictive multiplicity in classification. Advances in Neural Information Processing Systems, 35:28988–29000, 2022. Hsiang Hsu, Guihong Li, Shaohan Hu, and Chun-Fu Chen. Dropout-based Rashomon set exploration for efficient predictive multiplicity estimation. In International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=Sf2A2PUXO3. Keller Jordan. Calibrated chaos: Variance between runs of neural network training is harmless and inevitable. arXiv preprint arXiv:2304.01910, 2023. Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 6405–6416, 2017. Michel Ledoux. The concentration of measure phenomenon, volume 89 of Mathematical Surveys and Monographs. American Mathematical Society, 2001. Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David Crandall, and Dhruv Batra. Why M heads are better than one: Training a diverse ensemble of deep networks. arXiv preprint arXiv:1511.06314, 2015. Francesco Leofante, Elena Botoeva, and Vineet Rajani. Counterfactual explanations and model multiplicity: a relational verification view. In Proceedings of the 20th International Conference on Principles of Knowledge Representation and Reasoning, pp. 763–768, 8 2023. doi: 10.24963/kr.2023/78. URL https://doi.org/10.24963/kr.2023/78. Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. In Advances in Neural Information Processing Systems, volume 35, 2022. URL https://proceedings.neurips. cc/paper_files/paper/2022/hash/0cde695b83bd186c1fd456302888454c-Abstract-Conference. html. Charles Marx, Flavio Calmon, and Berk Ustun. Predictive multiplicity in classification. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pp. 6765–6774. PMLR, 2020. URL https://proceedings.mlr.press/v119/marx20a.html. Colin McDiarmid. On the method of bounded differences. In Surveys in Combinatorics, 1989, volume 141 of London Mathematical Society Lecture Note Series, pp. 148–188. Cambridge University Press, 1989. doi: 10.1017/CBO9781107359949.008. URL https://doi.org/10.1017/CBO9781107359949.008. Gábor Melis, Chris Dyer, and Phil Blunsom. On the state of the art of evaluation in neural language models. In International Conference on Learning Representations, 2018. URL https://openreview. net/forum?id=ByJHuTgA-.

24

S. Moro, P. Rita, and P. Cortez. Bank marketing. UCI Machine Learning Repository, 2014. DOI: https://doi.org/10.24432/C5K306. Jeremy Nixon, Balaji Lakshminarayanan, and Dustin Tran. Why are bootstrapped deep ensembles not better? In ”I Can’t Believe It’s Not Better!”NeurIPS 2020 workshop, 2020. NVIDIA Corporation. NGC catalog user guide. https://docs.nvidia.com/ngc/latest/ ngc-catalog-user-guide.html, 2026. Accessed August 2026. Jon Perez-Cerrolaza, Jaume Abella, Markus Borg, Carlo Donzella, Jesús Cerquides, Francisco J Cazorla, Cristofer Englund, Markus Tauber, George Nikolakopoulos, and Jose Luis Flores. Artificial intelligence for safety-critical systems in industrial and transportation domains: A survey. ACM Computing Surveys, 56(7):1–40, 2024. Eric D Peterson. Machine learning, predictive analytics, and clinical practice: can the past inform the present? JAMA, 322(23):2283–2284, 2019. David Picard. Torch. manual seed (3407) is all you need: On the influence of random seeds in deep learning architectures for computer vision. arXiv preprint arXiv:2109.08203, 2021. Sebastian Pineda Arango, Maciej Janowski, Lennart Purucker, Arber Zela, Frank Hutter, and Josif Grabocka. Ensembling finetuned language models for text classification. In NeurIPS 2024 Workshop on Fine-Tuning in Modern Machine Learning: Principles and Scalability, 2024. URL https: //arxiv.org/abs/2410.19889. Rahul Rahaman and Alexandre Thiery. Uncertainty quantification and deep ensembles. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 20063–20075. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/file/ a70dc40477bc2adceef4d2c90f47eb82-Paper.pdf. Nils Reimers and Iryna Gurevych. Reporting score distributions makes a difference: Performance study of LSTM-networks for sequence tagging. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 338–348, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/v1/D17-1035. URL https://aclanthology.org/D17-1035/. Hadi Salman, Jerry Li, Ilya Razenshteyn, Pengchuan Zhang, Huan Zhang, Sebastien Bubeck, and Greg Yang. Provably robust deep learning via adversarially trained smoothed classifiers. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedings. neurips.cc/paper_files/paper/2019/file/3a24b25a7b092a252166a1641ae953e7-Paper.pdf. Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=9Vrb9D0WI4.

25

Lesia Semenova, Cynthia Rudin, and Ronald Parr. On the existence of simpler machine learning models. In 2022 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’22. ACM, June 2022. doi: 10.1145/3531146.3533232. URL http://dx.doi.org/10.1145/3531146.3533232. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In David Yarowsky, Timothy Baldwin, Anna Korhonen, Karen Livescu, and Steven Bethard (eds.), Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631– 1642, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL https: //aclanthology.org/D13-1170/. Cecilia Summers and Michael J. Dinneen. Nondeterminism and instability in neural network optimization. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pp. 9913–9922. PMLR, 2021. URL https://proceedings.mlr.press/v139/summers21a.html. Rutgers University. Rutgers office of advance research computing. resources/amarel/, 2026.

https://oarc.rutgers.edu/

Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge University Press, 2018. Xi Wang, Laurence Aitchison, and Maja Rudolph. LoRA ensembles for large language model fine-tuning, 2023. URL https://arxiv.org/abs/2310.00035. Jamelle Watson-Daniels, David C Parkes, and Berk Ustun. Predictive multiplicity in probabilistic classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 10306–10314, 2023. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Qun Liu and David Schlangen (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38–45, Online, October 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-demos.6. URL https://aclanthology. org/2020.emnlp-demos.6/. Rui Xin, Chudi Zhong, Zhi Chen, Takuya Takagi, Margo Seltzer, and Cynthia Rudin. Exploring the whole Rashomon set of sparse decision trees. Advances in Neural Information Processing Systems, 35: 14071–14084, 2022. Ming Yuan and Marten Wegkamp. Classification methods with reject option based on convex risk minimization. Journal of Machine Learning Research, 11(5):111–130, 2010. URL https://www.jmlr.org/ papers/v11/yuan10a.html. Xu-Yao Zhang, Guo-Sen Xie, Xiuli Li, Tao Mei, and Cheng-Lin Liu. A survey on learning to reject. Proceedings of the IEEE, 111(2):185–215, 2023. doi: 10.1109/JPROC.2023.3238024. Yuqicheng Zhu, Nico Potyka, Mojtaba Nayyeri, Bo Xiong, Yunjie He, Evgeny Kharlamov, and Steffen Staab. Predictive multiplicity of knowledge graph embeddings in link prediction. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 334–354, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-emnlp.19. URL https://aclanthology.org/2024. findings-emnlp.19/. 26

Liu Zhuang, Lin Wayne, Shi Ya, and Zhao Jun. A robustly optimized BERT pre-training approach with post-training. In Sheng Li, Maosong Sun, Yang Liu, Hua Wu, Kang Liu, Wanxiang Che, Shizhu He, and Gaoqi Rao (eds.), Proceedings of the 20th Chinese National Conference on Computational Linguistics, pp. 1218–1227, Huhhot, China, August 2021. Chinese Information Processing Society of China. URL https://aclanthology.org/2021.ccl-1.108/.

A

Appendix

A.1

Notation Summary

We recap the notation used in the proofs in Table 3.

A.2

Measure concentration tools

We restate McDiarmid’s inequality (McDiarmid, 1989) in the form in which we use it to make the manuscript more self-contained. Lemma 1 (McDiarmid’s inequality (McDiarmid, 1989)). Let z1 , . . . , zN be independent random variables taking values in a set Z. Let Φ : Z N → R be a function of (z1 , . . . , zN ). Assume Φ satisfies the bounded differences property: for each i ∈ {1, . . . , n} there exists ci ≥ 0 such that for all z1 , . . . , zn , zi′ ∈ Z, Φ(z1 , . . . , zi , . . . , zN ) − Φ(z1 , . . . , zi′ , . . . , zN ) ≤ ci .

(22)

Then for every t > 0, 2t2 Pz (|Φ(z1 , . . . , zN ) − E[Φ(z1 , . . . , zN )]| ≥ t) ≤ 2 exp − PN 2 i=1 ci

! (23)

Lemma 2 (Hoeffding’s Inequality). Let z1 , z2 , . . . , zN be independent random variables such that for each i, zi ∈ [a, b] almost surely. Then for any ε > 0, we have !   N 1 X 2N ε2 Pz zi − E[zi ] ≥ ε ≤ 2 exp − . (24) N (b − a)2 i=1

Proof. We refer the readers to Ledoux (2001) for proof of this lemma.

A.3

Probabilistic guarantee on our proposed consistency measure

Before proving our main results, we state and prove some lemmas that we will leverage. All our proofs make some common assumptions which we restate below. Assumption 1 (Rashomon consistency assumptions). The set R = {fk : k ∈ [M ]} is an i.i.d. sample from a distribution τ on R. Every model f in the Rashomon set R is η-locally Lipschitz as in Definition 5. For fixed parameters β, σ and ϵ2 , x belongs to the subset of test instances for which the Rashomon set is (β, σ)-consistent according to Definition 3, and ϵ2 -stable according to Definition 4. We denote this set cons (β, σ, ϵ ) = {x ∈ D as Dtest 2 test : cσ (R, x) ≥ β and R is ϵ2 stable at x} ⊆ Dtest . Lemma 3. Let R, R, and x satisfy Assumption 1 for parameter ϵ2 > 0. Let f¯(R, x) be the ensemble model defined in Equation (6). Define   1 1 . gmarg (R, x) = f¯(R, x) − − ER f¯(R, x) − 2 2

(25)

  M ϵ24 PR (gmarg (R, x) ≥ ϵ4 ) ≤ 2 exp − 2 . 2ϵ2

(26)

Then for all ϵ4 > 0,

27

cons . Since R is sampled i.i.d. according to τ , we will use McDiarmid’s Proof. Fix ϵ2 > 0 and x ∈ Dtest inequality (Lemma 1) on the ensemble prediction f¯(R, x). To do so, we must verify the bounded differences property in (22). Let R = {f1 , . . . , fk , . . . , fM } ⊆ R and R′ = R \ {fk } ∪ {fk′ }, where fk′ ∈ R. Using triangle inequality and the fact that R is ϵ2 -stable at x (Definition 4):

f¯(R, x) − f¯(R′ , x) =

M

M

k=1

k=1

1 X 1 X ′ fk (x) − fk (x) M M

1 fk (x) − fk′ (x) = M 1 1 ′ ≤ |fk (x) − Ef [f (x)]| + f (x) − Ef [f (x)] M M k 2ϵ2 . ≤ M

(27)

2 Now applying Lemma 1 with ck = 2ϵ M , we have

   
−M ϵ24 . PR f¯(R, x) − ER f¯(R, x) ≥ ϵ4 ≤ 2 exp 2ϵ22 Let gmarg (R, x) be defined as in (25). From the reverse triangle inequality,   gmarg (R, x) ≤ f¯(R, x) − ER f¯(R, x) .   Therefore the event {gmarg (R, x) ≥ ϵ4 } implies { f¯(R, x) − ER f¯(R, x) ≥ ϵ4 }, so   −M ϵ24 PR (gmarg (R, x) ≥ ϵ4 ) ≤ 2 exp . 2ϵ22

(28)

(29)

(30)

Our main result involves showing measure concentration over perturbations and ensemble models. Since both are generated randomly, we are interested in the random variables corresponding to the differences δj,k (x) = |fk (x) − fk (vj )|

j ∈ [S], k ∈ [M ],

(31)

where R = {fk : k ∈ [M ]} are drawn i.i.d. from R and V = {vj : j ∈ [S]} are draw uniformly from (d) S2 (x, σ). Define M

∆(x) =

S

1 X1X δj,k (x). M S

(32)

j=1

k=1

∆(x) = ER,V [∆(x)].

(33)

We will analyze the concentration of these quantities against their expectations. Lemma 4. Let R, R, and x satisfy Assumption 1 for parameters σ > 0. Let V = {vj : j ∈ [S]} be a i.i.d.

(d)

random set of perturbations of x, distributed ∼ Unif(S2 (x, σ)). Define   M S X X 1 1 hA (R, x) = EV  |fk (x) − fk (vj )|. M S k=1

(34)

j=1

Then for all ϵ3 > 0 PR hA (R, x) − ∆(x) ≥ ϵ3 28

 −2M ϵ23 ≤ 2 exp . η2σ2 

(35)

(d)

Proof. Since each element of R is locally Lipschitz in the neighborhood S2 (x, σ) with constant η, for every perturbation vj , |fk (x) − fk (vj )| ≤ η∥x − vj ∥2 ≤ ησ.

(36)

This in turn means,  EV 

S 1X

S

 |fk (x) − fk (vj )| ≤ ησ.

(37)

j=1

h P S 1

i Thus, EV S j=1 |fk (x) − fk (vj )| ∈ [0, ησ] is an i.i.d. random variable since {fk : k ∈ [M ]} are i.i.d.. Applying Hoeffding’s inequality (Lemma 2) over this randomness, we have  
−2M ϵ23 PR hA (R, x) − ∆(x) ≥ ϵ3 ≤ 2 exp . (38) η2σ2 as desired. Lemma 5. Let R, R, and x satisfy Assumption 1 for parameters σ > 0. Let V = {vj : j ∈ [S]} be a i.i.d.

(d)

random set of perturbations of x, distributed ∼ Unif(S2 (x, σ)). Then for any ϵ4 > 0,    
−M ϵ24 −Sϵ24 PR,V ∆(x) − ∆(x) ≥ ϵ4 ≤ 2 exp + 2 exp 2η 2 σ 2 2η 2 σ 2

(39)

Proof. Let gvar (R, V, x) = ∆(x) − ∆(x) be the random deviation of the empirical variability term from its population counterpart. We decompose this gap as follows: gvar (R, V, x) ≤ |∆(x) − hA (R, x)| + hA (R, x) − ∆(x) . | {z } | {z } gA (R,V,x)

(40)

gB (R,x)

Lemma 4 shows that for any ϵ3 > 0  −2M ϵ23 PR (gB (R, x) ≥ ϵ3 ) ≤ 2 exp . η2σ2 

(41)

We now argue that gA (R, V, x) is also bounded with high probability. Let R = {fk : k ∈ [M ]} be a realization of R and let M

zj =

1 X |fk (x) − fk (vj )|. M

(42)

k=1

The random variables zj are i.i.d. since {vj : j ∈ [S]} are i.i.d.. Since |fk (x) − fk (vj )| ≤ ησ, applying Hoeffding’s inequality to {zj : j ∈ [S]} yields     S X 1 −2Sϵ23   PV zj − hA (R, x) ≥ ϵ3 ≤ 2 exp (43) S η2σ2 j=1

This is an upper bound on gA (R, V, x). Since Equation (43) holds uniformly for all R = R, applying the law of total probability yields,   X −2Sϵ23 PR,V [|gA (R, V, x)| ≥ ϵ3 ] = PV|R=R [gA (R, V, x) ≥ ϵ3 |R = R]P[R = R] ≤ 2 exp , (44) η2σ2 R

29

Then setting ϵ3 = ϵ4 /2 we have PR,V (gvar (R, V, x) ≥ ϵ4 ) ≤ PR,V (gA (R, V, x) ≥ ϵ4 /2, gB (R, x) ≥ ϵ4 /2)

(45)

≤ PR,V (gA (R, V, x) ≥ ϵ4 /2) + PR,V (gB (R, x) ≥ ϵ4 /2)     −M ϵ24 −Sϵ24 + 2 exp . ≤ 2 exp 2η 2 σ 2 2η 2 σ 2 Here we used the fact that for gvar (R, V, x) ≥ ϵ4 to hold, at least one of the terms gA (R, V, x) and gB (R, x) should be larger than ϵ4 /2. A.3.1

Proof of main results

Theorem 1 bounds the probability that the consistency score computed using a finite retail ensemble falls significantly below the corresponding consistency score of the Rashomon set. Let gmeasure (R, V, x) = |cσ (R, V, x) − cσ (R, x)|,

(46)

P 1 PM where cσ (R, V, x) = f¯(R, x) − 12 − S1 Sj=1 M k=1 |fk (x) − fk (vj )| from Definition 6 and cσ (R, x) = i.i.d.

Ef [f (x)] − 12 − Ef ,v [|f (x) − f (v)|]. Since the models in R are ∼ τ and the perturbations in V are h P i   PM i.i.d. (d) |fk (x) − fk (vj )| . ∼ Unif(S (x, σ)), equivalently, cσ (R, x) = ER f¯(R, x) − 1 − ER,V 1 S 1 2

2

S

j=1 M

k=1

cons , by Definition 3 of (β, σ) consistency of x, c (R, x) ≥ β. Therefore, if we can bound the Since x ∈ Dtest σ deviation gmeasure (R, V, x) as gmeasure (R, V, x) ≤ ϵ1 , we can bound cσ (R, V, x) as,

cσ (R, V, x) ≥ cσ (R, x) − ϵ1 ≥ β − ϵ1 .

(47)

Thus, to show cσ (R, V, x) ≥ β − ϵ1 , it suffices to upper bound the term gmeasure (R, V, x) which we prove next. Theorem 1 (Consistency guarantee). Let R, R, and x satisfy Assumption 1 for parameters β ∈ [− 12 , 12 ], cons , let V = {v : j ∈ [S]} be a random set of perturbations of x, and σ, ϵ5 > 0. For a test instance x ∈ Dtest j i.i.d. (d) ¯ distributed ∼ Unif(S2 (x, σ)), and let f (R, x) be the ensemble prediction at x as in Equation (6). Given P 1 PM the consistency measure cσ (R, V, x) = f¯(R, x) − 12 − S1 Sj=1 M k=1 fk (x) − fk (vj ) from Definition 6, for all ϵ1 ≥ 0,       −M ϵ21 −Sϵ21 −M ϵ21 PR,V [cσ (R, V, x) ≥ β − ϵ1 ] ≥ 1 − 2 exp − 2 exp − 2 exp . 8η 2 σ 2 8η 2 σ 2 8ϵ22

(15)

Proof. Since gmeasure (R, V, x) = |cσ (R, V, x) − cσ (R, x)|, we have gmeasure (R, V, x) =   S M S M X X X X   1 1 1 1 1 1 f¯(R, x) − − |fk (x) − fk (vj )| − ER f¯(R, x) − + ER,V  |fk (x) − fk (vj )| . 2 S M 2 S M j=1

j=1

k=1

k=1

(48) After rearranging the terms as in Equation (48), we represent them with two functions for notation convenience. The function   1 1 gmarg (R, x) = f¯(R, x) − − ER f¯(R, x) − , 2 2 30

(49)

represents the difference between the margin of a single ensemble and the margin of the expected ensemble. The other term   S M S M X X X X 1 1 1 1 gvar (R, V, x) = |fk (x) − fk (vj )| − ER,V  |fk (x) − fk (vj )| , (50) S M S M j=1

j=1

k=1

k=1

captures how predictions vary when we perturb the input x. The function compares the average local stability of an ensemble to its population counterpart. From the triangle inequality we have, gmeasure (R, V, x) ≤ gmarg (R, x) + gvar (R, V, x).

(51)

  −M ϵ2 From Lemma 3, PR [gmarg (R, x) ≥ ϵ4 ] ≤ 2 exp 2ϵ2 4 , and from Lemma 5, PR,V [gvar (R, V, x) ≥ ϵ4 ] ≤ 2     −M ϵ24 −Sϵ24 2 exp 2η2 σ2 + 2 exp 2η2 σ2 . For gmeasure (R, V, x) to be greater than ϵ1 , at least one of the terms gmarg (R, x) or gvar (R, V, x) should be greater than ϵ21 . Fixing ϵ4 = ϵ21 , we get h h ϵ1 i ϵ1 i PR,V [gmeasure (R, V, x) ≥ ϵ1 ] ≤ PR gmarg (R, x) ≥ + PR,V gvar (R, V, x) ≥ . 2 2

(52)

Thus, PR,V [gmeasure (R, V, x) ≥ ϵ1 ] ≤ 2 exp

 −M ϵ2  1

8ϵ22

   −M ϵ2  −Sϵ21 1 + 2 exp . + 2 exp 8η 2 σ 2 8η 2 σ 2

(53)

It follows, PR,V [gmeasure (R, V, x) ≤ ϵ1 ] ≥ 1 − 2 exp

 −M ϵ2  1

8ϵ22

   −M ϵ2  −Sϵ21 1 − 2 exp − 2 exp . 8η 2 σ 2 8η 2 σ 2

(54)

   −M ϵ2  −Sϵ21 1 − 2 exp . − 2 exp 8η 2 σ 2 8η 2 σ 2

(55)

When combined with Equation (47) yields, PR,V [cσ (R, V, x) ≥ β − ϵ1 ] ≥ 1 − 2 exp

A.4

 −2M ϵ2  1

8ϵ22

Local consistency

We also provide a consistency guarantee using a related measure that assesses the average local margin of an ensemble of models and penalizes it with the same variability term as before. We call this measure the local consistency (Definition 10). Under this new measure we define consistent test points with respect to the empirical Rashomon set R in Definition 9, and show that the ensemble of a retail set R will also remain consistent at these test points with high probability (exponential in the number of samples S from a local neighborhood around x and the size M of the retail set). i.i.d.

Definition 9 ((β, σ)-local consistency of R). Given R with distribution τ , a random model f ∼ τ , an i.i.d.

(d)

instance x ∈ Dtest , a random perturbation v ∼ Unif(S2 (x, σ)), the local expected prediction Ef ,v [f (v)] at x, and parameters β ∈ [− 12 , 21 ] and σ > 0, let lmcσ (R, x) = Ef ,v [f (v)] − 12 − Ef ,v [|f (x) − f (v)|], the local expectation of the ensemble margin of R at x, penalized by the expected ensemble variability around x . Then given x ∈ X , the ensemble of R is said to be (β, σ)-locally consistent at x if lmcσ (R, x) ≥ β.

31

(56)

Definition 10 ((β − ϵ1 , σ)-local consistency of a retail set). Given a Rashomon retail set R = {fk : k ∈ (d) [M ]} ⊆ R of size M , a set of perturbed samples V = {vj : j ∈ [S]} ⊆ S2 (x, σ), the ensemble prediction f¯(R, V ) in Definition 2, and a constant ϵ1 > 0, let lmcσ (R, V, x) be defined as S

lmcσ (R, V, x) = f¯(R, V ) −

M

1X 1 X 1 − fk (x) − fk (vj ) . 2 S M j=1

(57)

j=1

for a fixed test instance x. Then, R is said to be (β − ϵ1 , σ)-locally consistent if, lmcσ (R, V, x) ≥ β − ϵ1

(58)

Similar to the conditions needed to establish the theoretical guarantees for our proposed consistency measure, cσ (R, V, x), we require local stability of model predictions in the Rashomon set R along with Lipshchitz continuity as in Definition 5. Definition 11 (ϵ5 local stability of R at x). Given a Rashomon set R, a test point x ∈ Dtest , a perturbed (d) sample of x, v ∈ S2 (x, σ), and a small constant ϵ5 > 0, R is said to be locally ϵ5 stable if, (d)

|f (v) − Ef [f (v)]| ≤ ϵ5 , for all f ∈ R, v ∈ S2 (x, σ).

(59)

We list the assumptions required to establish a probabilistic guarantee on lmcσ below. Assumption 2 (Rashomon local consistency assumptions). The set R = {fk : k ∈ [M ]} is an i.i.d. sample from a distribution τ on R. Every model f in the Rashomon set R is η-locally Lipschitz as defined in 5. For fixed parameters β, σ and ϵ5 , x belongs to the subset of test instances which are (β, σ)-locally consistent according to Definition 9, and ϵ5 stable according to Definition 11. We denote this set as localcons (β, σ, ϵ ) = {x ∈ D Dtest 5 test : lmcσ (R, x) ≥ β and R is ϵ5 locally stable at x}. localcons . For notational simplicity, The concentration results that follow are stated only for a fixed x ∈ Dtest localcons on β, σ, ϵ5 . we suppress the dependence Dtest localcons , Theorem 2 provides the following probabilistic guarantee that the local For a fixed x ∈ Dtest consistency measure on the ensemble of the retail set lmcσ (R, V, x) will not deviate too much from the local consistency measure on the empirical Rashomon set lmcσ (R, x).

Theorem 2 (Local consistency guarantee). Let R, R, and x satisfy Assumption 2 for parameters localcons , let V = {v : j ∈ [S]} be a random set of perβ ∈ [− 21 , 12 ], and σ, ϵ5 > 0. For a test instance x ∈ Dtest j i.i.d. (d) turbations of x, distributed ∼ Unif(S2 (x, σ)), and let f¯(R, V ) be the ensemble prediction as in Definition P 1 PM (2). Given the local consistency measure lmcσ (R, V, x) = f¯(R, V ) − 21 − S1 Sj=1 M k=1 fk (x) − fk (vj ) from Definition 10, for all ϵ1 ≥ 0,        −M ϵ2  −M ϵ21 −Sϵ21 −Sϵ21 1 PR,V [lmcσ (R, V, x) ≥ β − ϵ1 ] ≥ 1 − 2 exp − 2 exp − 2 exp − 2 exp . 32η 2 σ 2 8η 2 σ 2 8η 2 σ 2 32ϵ25 (60) Proof. We refer the readers to Section A.4.2 for the proof. A.4.1

Probabilistic guarantee on local ensemble consistency

Our main result in this section is a concentration bound for the average of perturbed predictions from an ensemble model. Before proving the probabilistic guarantee on the local consistency measure lmcσ (R, V, x), we first prove the following lemmas that we will leverage for our proof. 32

Lemma 6. Let R, R, and x satisfy Assumption 2 for parameter ϵ5 > 0. Let V = {vj : j ∈ [S]} be i.i.d. (d) a random set of perturbations of x, distributed ∼ Unif(S (x, σ)), and f¯(R, V ) be the ensemble model 2

defined in 2. Define   1 hB (R, x) = EV f¯(R, V) − , 2   1 Γ(x) = ER,V f¯(R, V) − 2

(61) (62)

Then PR hB (R, x) − Γ(x) ≥ ϵ3

 −M ϵ23 . ≤ 2 exp 2ϵ25 

(63)

localcons . Since R = {f : k ∈ [M ]} is sampled i.i.d. according to τ , we will use Proof. Fix ϵ5 > 0 and x ∈ Dtest k McDiarmid’s inequality (Lemma 1) on the expected local prediction of the ensemble, EV [f¯(R, V)]. To do so, we must verify the bounded differences property in Equation (22). Let R = {f1 , . . . , fk , . . . , fM } ⊆ R and R′ = R \ {fk } ∪ {fk′ }, where fk′ ∈ R. Using the triangle inequality and the fact that R is ϵ5 -stable at x (Definition 11)

EV [f¯(R, V)] − EV [f¯(R′ , V) =

M 1 X

M

k=1

 EV 

S 1X

S

 fk (vj ) −

j=1

M 1 X

M

k=1

 EV 

S 1X

S

 fk′ (vj )

(64)

j=1

  1 Ev [fk (v)] − Ev fk′ (v) M   1 1 ≤ |Ev [fk (v)] − Ef ,v [f (v)]| + Ev fk′ (v) − Ef ,v [f (v)] M M ϵ5 ϵ5 + ≤ M M 2ϵ5 = . M =

(65)

(66)

2 Now applying Lemma 1 with ck = 2ϵ M , we have

PR

   
−M ϵ23 ¯ ¯ EV [f (R, V)] − ER,V f (R, V) ≥ ϵ3 ≤ 2 exp . 2ϵ25

Let hB , and Γ be defined as in Equation (61),and (62). From the reverse triangle inequality,   hB (R, x) − Γ(x) ≤ EV [f¯(R, V)] − ER,V f¯(R, V) .   Therefore, the event { hB (R, x) − Γ(x) ≥ ϵ3 } implies { EV [f¯(R, V)] − ER,V f¯(R, V) ≥ ϵ3 }, so  
−M ϵ23 PR hB (R, x) − Γ(x) ≥ ϵ3 ≤ 2 exp . 2ϵ25

(67)

(68)

(69)

Lemma 7. Let R, R, and x satisfy Assumption 1 for parameters σ, ϵ5 > 0. Let V = {vj : j ∈ [S]} be i.i.d. (d) a random set of perturbations of x, distributed ∼ Unif(S (x, σ)), and f¯(R, V ) be the ensemble model 2

defined in 2. Define 1 Γ(x) = f¯(R, V) − . 2 33

(70)

Then for any ϵ4 > 0, PR,V Γ(x) − Γ(x) ≥ ϵ4

−Sϵ24 ≤ 2 exp 8η 2 σ 2 



  −M ϵ24 . + 2 exp 8ϵ25

(71)

Proof. Our goal is to show concentration for the local margin term Γ(x) to Γ(x). The variability comes from two distinct sources of randomness, • The randomness in the perturbation samples, V = {vj : j ∈ [S]} when R is fixed. • The randomness in the retail set R = {fk : k ∈ [M ]}. Let glmarg (R, V, x) = Γ(x) − Γ(x) . We can thus decompose the gap as follows glmarg (R, V, x) ≤ |Γ(x) − hB (R, x)| + hB (R, x) − Γ(x) . | {z } | {z } gC (R,V,x)

(72)

gD (R,x)

Lemma 6 shows that for any ϵ3 ,  −M ϵ23 . PR (gD (R, x) ≥ ϵ3 ) ≤ 2 exp 2ϵ25 

(73)

We now argue that gC (R, V, x) = |glmarg (R, V, x) − hB (R, x)| is also bounded with high probability. Let R = {fk : k ∈ [M ]} be a realization of R and let f¯(R, V, x) be deterministic in R. Since V = {vj : j ∈ [S]} i.i.d.

(d)

is sampled ∼ Unif(S2 (x, σ)) we will use McDiarmid’s inequality (Lemma 1) on the ensemble prediction f¯(R, V). To do so we must verify the bounded differences property in (22). Let V = {v1 , . . . , vj , . . . , vS } ⊆ (d) (d) S2 (x, σ) and V ′ = V \ {vj } ∪ {vj′ }, where vj′ ∈ S2 (x, σ). Using triangle inequality and the fact every (d)

element of R is locally Lipschitz with constant η in the neighborhood S2 (x, σ), for every perturbation vj , S

S

j=1

j=1

1X¯ 1X¯ f¯(R, V ) − f¯(R, V ′ ) = f (R, vj ) − f (R, vj′ ) S S 1 ¯ f (R, vj ) − f¯(R, vj′ ) S M M 1 1 X 1 1 X ≤ |fk (v) − fk (x)| + fk (x) − fk (v ′ ) SM SM =

k=1

k=1

2ησ . ≤ S

(74)

Now applying Lemma 1 with ck = 2ησ S , we have PV

   
−Sϵ23 ¯ ¯ f (R, V) − EV f (R, V) ≥ ϵ3 ≤ 2 exp . 2η 2 σ 2

(75)

From the triangle inequality we have,   gC (R, V) ≤ f¯(R, V) − EV f¯(R, V, x) .   Therefore the event { f¯(R, V, x) − EV f¯(R, V, x) ≥ ϵ3 } implies {gC (R, V, x) ≥ ϵ3 }, so   −Sϵ23 PV (|gC (R, V, x)| ≥ ϵ3 ) ≤ 2 exp . 2η 2 σ 2 34

(76)

(77)

Since Equation (77) holds uniformly for all R = R, applying the law of total probability yeilds,   X −Sϵ23 , PR,V (|gC (R, V, x)| ≥ ϵ3 ) = PV|R=R (gC (R, V, x) ≥ ϵ3 |R = R) P(R = R) ≤ 2 exp 2η 2 σ 2

(78)

R

Then setting ϵ3 = ϵ4 /2 we have PR,V (glmarg (R, V, x) ≥ ϵ4 ) ≤ PR,V (gC (R, V, x) ≥ ϵ4 /2, gD (R, V, x) ≥ ϵ4 /2)

(79)

≤ PR,V (gC (R, V, x) ≥ ϵ4 /2) + PR,V (gD (R, x) ≥ ϵ4 /2)     −Sϵ24 −M ϵ24 . ≤ 2 exp + 2 exp 8η 2 σ 2 8ϵ25 Here we used the fact that for glmarg (R, V, x) ≥ ϵ4 to hold, at least one of the terms gC (R, V, x) and gD (R, x) should be larger than ϵ4 /2. A.4.2

Proof of Theorem 2

Theorem 2 bounds the probability that the local consistency score computed using a finite retail ensemble falls significantly below the corresponding consistency score of the full Rashomon Let P set. 1 PM |f (x) − fk ( glmeasure (R, V, x) = |lmcσ (R, V, x) − lmcσ (R, x)|, where lmcσ (R, V, x) = f¯(R, V, x) − 21 − S1 Sj=1 M k=1 i k h P   1 P M from Definition 10 and lmcσ (R, x) = ER,V f¯(R, V) − |fk (x) − fk (vj )| . − ER,V 1 S 1 2

S

j=1 M

k=1

localcons , by Definition 9 of (β, σ)-local consistency of x, lmc (R, x) ≥ β. Therefore, if Since x ∈ Dtest σ we can bound the deviation glmeasure (R, V, x) as glmeasure (R, V, x) ≤ ϵ1 , we can bound lmcσ (R, V, x) as,

lmcσ (R, V, x) ≥ lmcσ (R, x) − ϵ1 ≥ β − ϵ1 .

(80)

Thus to show cσ (R, V, x) ≥ β − ϵ1 , it suffices to upper bound the term gmeasure (R, V ) which we prove next. Theorem 2 (Local consistency guarantee). Let R, R, and x satisfy Assumption 2 for parameters localcons , let V = {v : j ∈ [S]} be a random set of perβ ∈ [− 12 , 12 ], and σ, ϵ5 > 0. For a test instance x ∈ Dtest j i.i.d. (d) turbations of x, distributed ∼ Unif(S2 (x, σ)), and let f¯(R, V ) be the ensemble prediction as in Definition P 1 PM (2). Given the local consistency measure lmcσ (R, V, x) = f¯(R, V ) − 21 − S1 Sj=1 M k=1 fk (x) − fk (vj ) from Definition 10, for all ϵ1 ≥ 0,        −M ϵ2  −Sϵ21 −M ϵ21 −Sϵ21 1 − 2 exp − 2 exp − 2 exp . PR,V [lmcσ (R, V, x) ≥ β − ϵ1 ] ≥ 1 − 2 exp 32η 2 σ 2 8η 2 σ 2 8η 2 σ 2 32ϵ25 (60) Proof. Given glmeasure (R, V, x) = |lmcσ (R, V, x) − lmcσ (R, x)|, we have glmeasure (R, V ) = S M 1X 1 X

1 f¯(R, V ) − − 2 S

j=1

M

 |fk (x) − fk (vj )| − ER,V f¯(R, V) − 

k=1



S M 1X 1 X

1 + ER,V  2 S

j=1

M

 |fk (x) − fk (vj )|

k=1

(81) After rearranging the terms, as in Equation (81), we represent them with two functions for notation convenience. The function   1 1 glmarg (R, V, x) = f¯(R, V ) − − ER,V f¯(R, V) − , 2 2 35

(82)

represents the difference between the average margin over perturbed samples of a single ensemble and the expected margin of the expected ensemble of models. The remaining terms   S M S M X X X X 1 1 1 1 glvar (R, V, x) = |fk (x) − fk (vj )| − ER,V  |fk (x) − fk (vj )| , (83) S M S M j=1

j=1

k=1

k=1

capture how predictions vary when we perturb the input x. From the triangle inequality, glmeasure (R, V, x) ≤ glmarg (R, V, x) + glvar (R, V, x). (84)     −Sϵ2 −M ϵ2 From Lemma 7, PR,V [glmarg (R, V, x) ≥ ϵ4 ] ≤ 2 exp 8η2 σ42 + 2 exp 8ϵ2 4 , and from Lemma 5, 5     −M ϵ23 −Sϵ24 PR,V [glvar (R, V, x) ≥ ϵ4 ] ≤ 2 exp 2η2 σ2 + 2 exp 2η2 σ2 . For glmeasure (R, V, x) to be greater than ϵ1 , at least one of the terms glmarg (R, V, x) and glvar (R, V, x) should be greater than ϵ21 . Fixing ϵ4 = ϵ21 , we get h h ϵ1 i ϵ1 i PR,V [glmeasure (R, V, x) ≥ ϵ1 ] ≤ PR glmarg (R, V, x) ≥ + PR,V glvar (R, V, x) ≥ . (85) 2 2 Thus, −M ϵ21 PR,V [glmeasure (R, V, x) ≥ ϵ1 ] ≤ 2 exp 32ϵ25 



     −M ϵ2  −Sϵ21 −Sϵ21 1 + 2 exp + 2 exp . + 2 exp 32η 2 σ 2 8η 2 σ 2 8η 2 σ 2 (86)

It follows that        −M ϵ2  −M ϵ21 −Sϵ21 −Sϵ21 1 PR,V [glmeasure (R, V, x) ≤ ϵ1 ] ≥ 1 − 2 exp − 2 exp − 2 exp − 2 exp . 32η 2 σ 2 8η 2 σ 2 8η 2 σ 2 32ϵ25 (87) When combined with Equation (80) yields,        −M ϵ2  −Sϵ21 −M ϵ21 −Sϵ21 1 − 2 exp − 2 exp . PR,V [lmcσ (R, V, x) ≥ β − ϵ1 ] ≥ 1 − 2 exp − 2 exp 2 2 2 2 2 2 2 32η σ 8η σ 8η σ 32ϵ5 (88)

A.5

Multiplicity metrics

As mentioned in the Introduction, predictive multiplicity means that models may disagree on an individual test instance x. When these models are in the Rashomon set, this disagreement may indicate that x is fundamentally “difficult” because even well-performing models disagree on it. Several measures have been proposed to quantify multiplicity. Definition 12 (Pairwise Disagreement (Black et al., 2022a)). The pairwise disagreement of the empirical Rashomon set R, at a test instance x, is the average disagreement between pairs of models fk , fl ∈ R and is defined as, pd(x) =

1 |R|(|R| − 1)

X hk ,hl ∈R,hk ̸=l

where h(x) = 1 f (x) ≥ 2 is the hard decision of f .
1

36

1(hk (x) ̸= hl (x)),

(89)

Definition 13 (Discrepancy (Marx et al., 2020)). The discrepancy of R, at a test instance x, is the fraction of conflicting predictions of models in R w.r.t. a reference model and is defined as, d(x) =

1 X 1(hk (x) ̸= h0 (x)), |R|

(90)

hk ∈R

  P 1 1 where we choose the reference model as h0 (x) = 1 |R| f (x) ≥ k fk ∈R 2 . We also look at multiplicity in the neighborhood of x using a new metric called the local discrepancy. Definition 14 (Local Discrepancy). The local discrepancy of R, at a test instance x, is the average frac(d) tion of conflicting predictions between models in R over samples v1 , . . . ., vS ∈ S2 (x, σ),and the prediction of a reference model at x, and is defined as, S

ldσ (x) =

1 X 1X 1(hk (vj ) ̸= h0 (x)), |R| S hk ∈R

(91)

j=1

  P 1 1 f (x) ≥ where we choose the reference model as h0 (x) = 1 |R| fk ∈R k 2 We also consider two multiplicity metrics: prediction variance and prediction range, which focus on predicted probabilities rather than the ‘hard’ predicted label. Definition 15 (Prediction Variance (Hamman et al., 2025; Watson-Daniels et al., 2023)). The prediction variance of R is the variance of model outputs for a given test instance x, and is defined as pv(x) =

1 X 1 X (fk (x) − fk (x))2 . |R| |R| fk ∈R

(92)

fk ∈R

Definition 16 (Prediction Range (Hamman et al., 2025; Watson-Daniels et al., 2023)). The prediction range of R is the range of predicted probabilities of models for a given test instance x, and is defined as, pr(x) = max fk (x) − min fk (x). fk ∈R

A.6

fk ∈R

(93)

Related measures

We define a few quantities that are related to our proposed consistency measure cσ (R, x) in Definition 6, and local consistency lmcσ (R, V, x) in Definition 10. Definition 17 (Margin). Given a Rashomon retail set R = {fk : k ∈ [M ]}, the ensemble model f¯(R, x) of size M , we define the margin of f¯(R, x) at a fixed instance x as, 1 mg(R, x) = f¯(R, x) − 2

(94)

Definition 18 (Local margin). Given a Rashomon retail set R = {fk : k ∈ [M ]} of size M , a set (d) V = {vj : j ∈ [S]} ⊆ S2 (x, σ) of S perturbed samples of x, and the local ensemble f¯(R, V ), we define the local margin of f¯(R, V ) at a fixed instance x as, 1 lmgσ (R, V, x) = f¯(R, V ) − 2

37

(95)

Definition 19 (Local variability). Given a Rashomon retail set R = {fk : k ∈ [M ]} of size M , and a (d) set V = {vj : j ∈ [S]} ⊆ S2 (x, σ) of S perturbed samples of x, we define the local variability at a fixed instance x as, lvσ (R, x) =

S

M

j=1

j=1

1X 1 X fk (x) − fk (vj ) S M

(96)

Definition 20 (HDMLD Local Stability Hamman et al. (2025)). Given a model from the Rashomon set (d) fk ∈ R, and a set V = {vj : j ∈ [S]} ⊆ S2 (x, σ) of S perturbed samples of x, we define the HDMLD local stability of the model at a fixed instance x as  S  1X fk (vj ) − fk (x) − fk (vj ) . hcσ (fk , x) = S

(97)

j=1

A.7

Additional Experiments

A.7.1

Additional plots showing high consistency thresholds substantially reduce audit risk for Rashomon warehouse

Section 4.2.2 presented the auditing results for the warehouse ensemble as a function of the threshold β for paraphrase detection on the MRPC dataset using ensembles of independently fine-tuned BERT models. Figures 5a and Figure 5b report the audit cost ratio, audit risk ratio, diversion fraction, and audit risk fraction curves from Section 4.2.2 for the remaining datasets and model architectures. Across all settings, we observe the same qualitative behavior as in Section 4.2.2. Increasing the audit threshold β makes the auditing system more conservative, substantially reducing audit risk before causing a sharp increase in audit cost and diversions. The vertical dashed line in each plot marks the value of threshold β for different datasets chosen according to the discussion in Section 4.2.2. A.7.2

Retail set audit outcomes match warehouse audit outcomes as M increases

Section 4.2.3 presented the auditing results for finite sized ensembles for paraphrase detection on the MRPC dataset using ensembles of independently fine-tuned BERT models. This appendix provides the corresponding results for the remaining datasets and model architectures. Figure 6 below report the audit cost ratio, audit risk ratio, diversion fraction, and audit risk fraction as functions of the retail ensemble size M , illustrating that the same convergence behaviour is consistently observed across all experimental settings. A.7.3

Understanding retail and warehouse audit disagreements

The results in section 4.2.3 compared retail and warehouse ensembles using aggregate auditing metrics from Definition 7, the audit cost ratio, audit risk ratio, diversion fraction, and audit risk fraction. While these metrics quantify the overall performance of the auditing system, they do not reveal how or why the retail and warehouse audit decisions differ. To better understand these differences, we next examine the combined audit outcomes introduced in Table 1. This allows us to identify whether disagreements arise from changes in audit cost, audit risk, or both relative to the baseline, and how these disagreement patterns evolve as the retail ensemble size increases. Table 4 summarizes how the audit outcomes produced by retail ensembles compare with those of the warehouse ensemble for paraphrase detection on the MRPC dataset using ensembles of fine-tuned BERT models. For M = 1, corresponding to no ensembling, nearly 73% of the test instances already fall into the baseline category, indicating that a single model agrees with the warehouse audit decision for the majority 38

(a) Audit cost ratio (ratiocost (RT , x)) versus audit risk ratio (ratiorisk (RT , x)). Results are shown (Left to Right) for: BERT on SST2, RoBERTa on MRPC, BIG-SCIENCE T0 on Bank, BIG-SCIENCE T0 on Adult. Increasing β decreases audit risk at the expense of higher audit cost.

(b) Diversion fraction (fracdiv (RT , x)) versus audit risk fraction (fracrisk (RT , x)). Results are shown (Left to Right) for: BERT on SST2, RoBERTa on MRPC, BIG-SCIENCE T0 on Bank, BIG-SCIENCE T0 on Adult. Increasing β increases the diversion fraction while reducing overall audit risk.

Figure 5: Comparing the audit performance metrics in Definition 7 for the warehouse ensemble model as a function of the threshold β used in the consistency check of Algorithm 1. The threshold β is chosen based on our discussion in Section 4.2.2. of inputs. At the same time, a very small fraction of test instances fall in the Neutral, Mixed, and Risk decreases category. Recall that both the mixed category and neutral category include test instances where the baseline and candidate models disagreed in their predicted labels. In the mixed category, the audit decisions for both the baseline and candidate models are incorrect: they either divert a correct prediction or automate an incorrect one. In the neutral category, on the other hand, the audit decisions for both the baseline and candidate models are correct: they either divert an incorrect prediction or automate a correct prediction. Across all experiments, we see that this combination of decisions disappears very quickly with only a small increase in ensemble size. We also note that more than 16.6% of the test instances fall into the Cost decreases category, compared with only 4.2% in the Cost increases category, indicating that a single model tends to audit fewer instances than the warehouse and therefore incurs lower audit cost. However, close to 5% test instances also fall in the Risk increases category. This indicates that the reduction in audit cost achieved without ensembling comes primarily at the expense of increased audit risk. As the retail ensemble size increases, the fraction of test instances in the Baseline category steadily rises from 73.1% for M = 1 to 90.9% for M = 30, demonstrating that retail audit outcomes increasingly match the warehouse audit outcomes with an increase in ensemble size. Correspondingly, the fractions of test instances assigned to all other categories decrease with increasing M . In particular, the Risk increases category decreases from almost 5% to close to 1%, even though this comes at the expense of the Cost decreases category, which decreases from 16.6% to 7.9%. These results show that while small retail ensembles benefit from lower audit costs, they also have a higher audit risk than the baseline, and increasing the ensemble size progressively recovers the warehouse auditing behavior. We observe a similar trend across all other architectures and datasets reported in Tables 5, 6, 7, and 8.

39

(a) Audit cost ratio (ratiocost ) versus audit risk ratio (ratiorisk ) as a function of the size of the retail ensemble. Results are shown (Left to Right) for: BERT on SST2, RoBERTa on MRPC, BIG-SCIENCE T0 on Bank, BIG-SCIENCE T0 on Adult. Increasing the ensemble size has a small impact in matching the warehouse once the ensemble is sufficiently large (around M= 10).

(b) Diversion fraction (fracdiv ) versus audit risk fraction (fracrisk ) as a function of the size of the retail ensemble. Results are shown (Left to Right) for: BERT on SST2, RoBERTa on MRPC, BIG-SCIENCE T0 on Bank, BIG-SCIENCE T0 on Adult. Diversion and risk fractions nearly match those of the warehouse, even with relatively modest ensemble sizes.

Figure 6: Comparing the audit metrics in Definition 7 for retail ensemble models as a function of the number of models M . For each size M , 500 ensembles of size M were generated by sampling with replacement from the warehouse set. Figures show the mean ± standard deviation (std) of the metrics in Definition 7 taken over the 500 retail ensembles. A.7.4

Visualizing relation between proposed consistency measure and different multiplicity metrics

Section 4.3 showed that the proposed consistency measure exhibits strong correlations with the multiplicity metrics computed using the Rashomon warehouse. We also looked at the behavior of hcσ (x) proposed by Hamman et al. (2025) and compared the behavior of this measure and our proposed consistency measure with respect to one multiplicity metric, pairwise disagreement, over randomly sampled test instances (Figure 4). Results were reported for a paraphrase detection task on the MRPC dataset using an ensemble of independently fine-tuned BERT models. Figure 7 and Figure 8 provide additional visualizations, for other multiplicity metrics. In Figure 7, we observe the same trend: the proposed consistency measure shows a strong negative correlation with all multiplicity metrics, which increases with retail ensemble size. As noted before, our measure assigns only high consistency scores to test instances with low multiplicity. On the other hand, hcσ (x) identifies several consistent instances as inconsistent by assigning low scores to some test instances with low multiplicity, as observed in Figure 8. A.7.5

Additional correlation tables from Section 4.3

In Section 4.3, we evaluated whether consistency measures computed from a finite retail ensemble do well in capturing predictive multiplicity metrics computed using the Rashomon warehouse. We compared the proposed measure with related measures like margin (mg(x), Definition 17), local consistency (lmcσ (c), 40

Definition 10), local margin (lmgσ (x), Definition 18), local variability (lvσ (x), Definition 19), and the consistency measure hcσ (x) of Hamman et al. (2025), using Spearman correlation with five warehouse multiplicity metrics: pairwise disagreement (Black et al., 2022a), discrepancy (Marx et al., 2020), local discrepancy, prediction variance, and prediction range (Hamman et al., 2025; Watson-Daniels et al., 2023). Tables 9a, 9b, 10a, and 10b extend this analysis to a paraphrase detection task on the MRPC dataset and sentiment classification on the SST2 dataset using independently fine-tuned RoBERTa models, and to a binary classification task on the Bank and Adult datasets using independently fine-tuned BIG-SCIENCE T0 models. Consistency measures are computed from a single retail ensemble of size M = 1, 5, 10, while multiplicity metrics are computed using the full Rashomon warehouse. Across these settings, we observe the same qualitative trends as in Section 4.3. The proposed consistency measure exhibits strong negative correlations with warehouse multiplicity metrics, and the magnitudes of these correlations increase with the ensemble size. Thus, higher consistency corresponds to lower predictive multiplicity, while lower consistency identifies instances with greater disagreement across the Rashomon warehouse. The strongest correlations are observed with prediction variance and prediction range, which capture variability in model confidence. The results also highlight the benefit of combining ensemble margin with local prediction variability. Measures based only on margin or local variability generally exhibit weaker correlations than the proposed consistency measure and local consistency, suggesting that both proximity to the decision boundary and local stability are important for characterizing predictive multiplicity.

41

Table 2: Absolute Spearman correlation between consistency measures and multiplicity metrics when applied to the (2a) paraphrase detection task on the MRPC dataset, and (2b) sentiment classification task on the SST2 dataset using an ensemble of independently fine-tuned BERT models. Higher values indicate greater agreement between the consistency measures (defined in Appendix A.6) computed for small retail ensembles and the multiplicity metrics (defined in Appendix A.5) computed from the entire Rashomon warehouse set. Rows highlighted in light blue correspond to the proposed consistency measures, cσ (x) and lmcσ (x). (a) MRPC

(b) SST2

Consistency measures Multiplicity metrics using using Rashomon warehouse retail pd(x) d(x) ldσ (x) pv(x) pr(x) ensembles M =1 cσ (x) 0.793 0.793 0.798 0.866 0.838 mg(x) 0.786 0.786 0.762 0.865 0.836 lmcσ (x) 0.755 0.755 0.778 0.841 0.816 lmgσ (x) 0.696 0.696 0.725 0.773 0.752 lvσ (x) 0.712 0.712 0.721 0.772 0.751 hcσ (x) 0.679 0.679 0.668 0.781 0.760 M =5 cσ (x) 0.885 0.885 0.891 0.957 0.929 mg(x) 0.851 0.851 0.821 0.927 0.898 lmcσ (x) 0.848 0.848 0.888 0.926 0.897 lmgσ (x) 0.814 0.814 0.864 0.896 0.868 lvσ (x) 0.851 0.851 0.881 0.928 0.903 hcσ (x) 0.713 0.713 0.691 0.823 0.804 M = 10 cσ (x) 0.893 0.893 0.897 0.964 0.937 mg(x) 0.884 0.884 0.875 0.957 0.928 lmcσ (x) 0.883 0.883 0.899 0.953 0.927 lmgσ (x) 0.850 0.850 0.869 0.926 0.898 lvσ (x) 0.887 0.887 0.883 0.956 0.932 hcσ (x) 0.660 0.660 0.636 0.777 0.762

Multiplicity metrics using Rashomon warehouse pd(x) d(x) ldσ (x) pv(x) pr(x) M =1 0.686 0.686 0.709 0.899 0.884 0.674 0.674 0.682 0.889 0.874 0.682 0.682 0.711 0.900 0.884 0.633 0.633 0.670 0.827 0.808 0.653 0.653 0.676 0.830 0.817 0.244 0.244 0.269 0.430 0.419 M =5 0.753 0.753 0.768 0.956 0.939 0.748 0.748 0.747 0.950 0.933 0.751 0.751 0.774 0.951 0.934 0.740 0.740 0.767 0.936 0.919 0.744 0.744 0.765 0.947 0.931 0.202 0.202 0.224 0.413 0.403 M = 10 0.762 0.762 0.776 0.971 0.956 0.758 0.758 0.762 0.970 0.953 0.758 0.758 0.780 0.964 0.950 0.751 0.751 0.776 0.957 0.942 0.758 0.758 0.777 0.961 0.948 0.187 0.187 0.199 0.390 0.383

42

Table 3: Table of notation used in this paper Symbol fθ hθ τ π Dtest N′ R(ϵ) R = {fk : k ∈ [M ]} f¯(R, x) (d)

S2 (x, σ) V = {vj : j ∈ [S]} f¯(R, V ) cσ (R, V, x) β ϵ1 η ϵ2 cons Dtest (β, σ, ϵ2 )

Meaning model from a set F parameterized by Θ thresholded model output 1(fθ ≥ 1/2) distribution on F induced by training data distribution on X × Y test dataset size of test data empirical Rashomon set with tolerance level ϵ i.i.d.

retail set with M models sampled ∼ τ ensemble model formed from the retail set Euclidean sphere of radius σ centered at x i.i.d.

(d)

set of perturbed samples of size S sampled ∼ UnifS2 (x, σ) average prediction of an ensemble of retail set on V consistency measure in Equation (13) consistency threshold deviation tolerance in the finite ensemble consistency guarantee Lipschitz constant of fθ stability tolerance of the Rashomon set Test instances that are β, σ consistent and ϵ2 stable.

43

Table 4: Audit outcomes and aggregated audit outcomes for a paraphrase detection task on the MRPC dataset using ensembles of independently fine-tuned BERT models. (a) Table reporting the fraction of test instances assigned to each outcome label o ∈ {0, . . . , 15} from Table 1. The baseline model is the ensemble of the Rashomon warehouse RT , with model error e(x; RT ), and audit decision r(x; RT ), while the candidate models are ensembles of q = {1, . . . , Q} random draws of the Rashomon retail set RM,q , with model error e(x; RM,q ) and audit decision r(x; RM,q ). The tuple (e(x; RT ), r(x; RT ), e(x; RM,q ), r(x; RM,q )) decides the combined audit outcomes, with label o ∈ {0, . . . , 15}. The table reports the mean ± standard deviation of the fraction of test instances assigned to each outcome label across q = {1, . . . , Q} random retail ensemble draws for different values of M . A desirable auditing strategy is one that minimizes excess audit costs and risks by increasing the Baseline category.

o

Meaning

M=1

M=3

M=5

M = 10

M = 30

0

Baseline

0.586 ± 0.012

0.585 ± 0.010

0.591 ± 0.008

0.596 ± 0.005

0.601 ± 0.002

1

Cost ↑

0.015 ± 0.011

0.018 ± 0.010

0.012 ± 0.008

0.007 ± 0.005

0.002 ± 0.002

2

Risk ↑

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

3

Neutral

0.002 ± 0.003

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

4

Cost ↓

0.133 ± 0.022

0.083 ± 0.019

0.076 ± 0.017

0.068 ± 0.014

0.062 ± 0.011

5

Baseline

0.082 ± 0.023

0.149 ± 0.020

0.161 ± 0.018

0.173 ± 0.015

0.183 ± 0.011

6

Risk ↑, Cost ↓

0.005 ± 0.006

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

7

Cost ↓

0.033 ± 0.011

0.020 ± 0.007

0.015 ± 0.006

0.011 ± 0.005

0.008 ± 0.004

8

Risk ↓

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

9

Risk ↓, Cost ↑

0.000 ± 0.001

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

10

Baseline

0.016 ± 0.001

0.016 ± 0.001

0.016 ± 0.001

0.017 ± 0.001

0.017 ± 0.001

11

Risk ↓

0.001 ± 0.001

0.001 ± 0.001

0.001 ± 0.001

0.000 ± 0.001

0.000 ± 0.001

12

Neutral

0.006 ± 0.005

0.000 ± 0.001

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

13

Cost ↑

0.027 ± 0.010

0.023 ± 0.007

0.019 ± 0.006

0.015 ± 0.006

0.009 ± 0.004

14

Risk ↑

0.047 ± 0.015

0.020 ± 0.008

0.016 ± 0.007

0.012 ± 0.006

0.010 ± 0.004

15

Baseline

0.047 ± 0.015

0.084 ± 0.010

0.092 ± 0.009

0.100 ± 0.007

0.108 ± 0.005

(b) Summary of the fraction of test instances assigned to each high-level audit category obtained by aggregating the corresponding outcome labels from Table 4a. Specifically, Baseline aggregates outcomes {0, 5, 10, 15}, Risk increases aggregates {2, 14}, Risk decreases aggregates {8, 11}, Cost increases aggregates {1, 13}, Cost decreases aggregates {4, 7}, Neutral aggregates {3, 12}, and Mixed aggregates {6, 9}. Category

M=1

M=3

M=5

M = 10

M = 30

Baseline Risk increases Risk decreases Cost increases Cost decreases Neutral Mixed

0.731 0.047 0.001 0.042 0.166 0.008 0.005

0.834 0.020 0.001 0.041 0.103 0.000 0.000

0.860 0.016 0.001 0.031 0.091 0.000 0.000

0.886 0.012 0.000 0.022 0.079 0.000 0.000

0.909 0.010 0.000 0.011 0.070 0.000 0.000

44

Table 5: Audit outcomes and aggregated audit outcomes for a sentiment classification task on the SST2 dataset using ensembles of independently fine-tuned BERT models. (a) Table reporting the fraction of test instances assigned to each outcome label o ∈ {0, . . . , 15} from Table 1. The baseline model is the ensemble of the Rashomon warehouse RT , with model error e(x; RT ), and audit decision r(x; RT ), while the candidate models are ensembles of q = {1, . . . , Q} random draws of the Rashomon retail set RM,q , with model error e(x; RM,q ) and audit decision r(x; RM,q ). The tuple (e(x; RT ), r(x; RT ), e(x; RM,q ), r(x; RM,q )) decides the combined audit outcomes, with label o ∈ {0, . . . , 15}. The table reports the mean ± standard deviation of the fraction of test instances assigned to each outcome label across q = {1, . . . , Q} random retail ensemble draws for different values of M . A desirable auditing strategy is one that minimizes excess audit costs and risks by increasing the Baseline category.

o

Meaning

M=1

M=3

M=5

M = 10

M = 30

0

Baseline

0.734 ± 0.008

0.739 ± 0.006

0.742 ± 0.004

0.749 ± 0.001

0.750 ± 0.000

1

Cost ↑

0.015 ± 0.008

0.011 ± 0.006

0.008 ± 0.004

0.001 ± 0.001

0.000 ± 0.000

2

Risk ↑

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

3

Neutral

0.001 ± 0.001

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

4

Cost ↓

0.080 ± 0.017

0.057 ± 0.012

0.051 ± 0.011

0.040 ± 0.005

0.040 ± 0.004

5

Baseline

0.077 ± 0.017

0.110 ± 0.012

0.120 ± 0.011

0.137 ± 0.005

0.138 ± 0.004

6

Risk ↑, Cost ↓

0.002 ± 0.002

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

7

Cost ↓

0.023 ± 0.007

0.014 ± 0.004

0.010 ± 0.003

0.004 ± 0.002

0.004 ± 0.002

8

Risk ↓

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

9

Risk ↓, Cost ↑

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

10

Baseline

0.012 ± 0.001

0.012 ± 0.001

0.012 ± 0.001

0.013 ± 0.000

0.013 ± 0.000

11

Risk ↓

0.001 ± 0.001

0.001 ± 0.001

0.001 ± 0.001

0.000 ± 0.000

0.000 ± 0.000

12

Neutral

0.001 ± 0.002

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

13

Cost ↑

0.011 ± 0.003

0.008 ± 0.003

0.006 ± 0.003

0.002 ± 0.001

0.001 ± 0.001

14

Risk ↑

0.018 ± 0.005

0.011 ± 0.003

0.010 ± 0.003

0.008 ± 0.002

0.008 ± 0.001

15

Baseline

0.026 ± 0.006

0.037 ± 0.004

0.040 ± 0.004

0.047 ± 0.002

0.048 ± 0.002

(b) Summary of the fraction of test instances assigned to each high-level audit category obtained by aggregating the corresponding outcome labels from Table 5a. Specifically, Baseline aggregates outcomes {0, 5, 10, 15}, Risk increases aggregates {2, 14}, Risk decreases aggregates {8, 11}, Cost increases aggregates {1, 13}, Cost decreases aggregates {4, 7}, Neutral aggregates {3, 12}, and Mixed aggregates {6, 9}. Category

M=1

M=3

M=5

M = 10

M = 30

Baseline Risk increases Risk decreases Cost increases Cost decreases Neutral Mixed

0.849 0.018 0.001 0.026 0.103 0.002 0.002

0.898 0.011 0.001 0.019 0.071 0.000 0.000

0.914 0.010 0.001 0.014 0.061 0.000 0.000

0.946 0.008 0.000 0.003 0.044 0.000 0.000

0.949 0.008 0.000 0.001 0.044 0.000 0.000

45

Table 6: Audit outcomes and aggregated audit outcomes for a paraphrase detection task on the MRPC dataset using ensembles of independently fine-tuned RoBERTa models. (a) Table reporting the fraction of test instances assigned to each outcome label o ∈ {0, . . . , 15} from Table 1. The baseline model is the ensemble of the Rashomon warehouse RT , with model error e(x; RT ), and audit decision r(x; RT ), while the candidate models are ensembles of q = {1, . . . , Q} random draws of the Rashomon retail set RM,q , with model error e(x; RM,q ) and audit decision r(x; RM,q ). The tuple (e(x; RT ), r(x; RT ), e(x; RM,q ), r(x; RM,q )) decides the combined audit outcomes, with label o ∈ {0, . . . , 15}. The table reports the mean ± standard deviation of the fraction of test instances assigned to each outcome label across q = {1, . . . , Q} random retail ensemble draws for different values of M . A desirable auditing strategy is one that minimizes excess audit costs and risks by increasing the Baseline category.

o

Meaning

M=1

M=3

M=5

M = 10

M = 30

0

Baseline

0.683 ± 0.018

0.692 ± 0.010

0.699 ± 0.007

0.706 ± 0.005

0.712 ± 0.002

1

Cost ↑

0.025 ± 0.016

0.021 ± 0.010

0.014 ± 0.007

0.007 ± 0.005

0.002 ± 0.002

2

Risk ↑

0.001 ± 0.001

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

3

Neutral

0.005 ± 0.004

0.000 ± 0.001

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

4

Cost ↓

0.090 ± 0.017

0.062 ± 0.014

0.055 ± 0.013

0.047 ± 0.011

0.039 ± 0.008

5

Baseline

0.061 ± 0.019

0.107 ± 0.018

0.120 ± 0.016

0.135 ± 0.013

0.149 ± 0.009

6

Risk ↑, Cost ↓

0.013 ± 0.009

0.001 ± 0.002

0.000 ± 0.001

0.000 ± 0.000

0.000 ± 0.000

7

Cost ↓

0.032 ± 0.008

0.026 ± 0.008

0.021 ± 0.007

0.014 ± 0.005

0.008 ± 0.004

8

Risk ↓

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

9

Risk ↓, Cost ↑

0.000 ± 0.001

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

10

Baseline

0.024 ± 0.003

0.025 ± 0.002

0.025 ± 0.002

0.026 ± 0.001

0.027 ± 0.000

11

Risk ↓

0.003 ± 0.002

0.002 ± 0.002

0.002 ± 0.002

0.001 ± 0.001

0.000 ± 0.000

12

Neutral

0.007 ± 0.005

0.001 ± 0.002

0.000 ± 0.001

0.000 ± 0.000

0.000 ± 0.000

13

Cost ↑

0.013 ± 0.005

0.013 ± 0.005

0.012 ± 0.005

0.009 ± 0.004

0.005 ± 0.003

14

Risk ↑

0.023 ± 0.006

0.012 ± 0.005

0.009 ± 0.004

0.006 ± 0.003

0.004 ± 0.002

15

Baseline

0.021 ± 0.008

0.038 ± 0.007

0.043 ± 0.007

0.049 ± 0.005

0.055 ± 0.004

(b) Summary of the fraction of test instances assigned to each high-level audit category obtained by aggregating the corresponding outcome labels from Table 6a. Specifically, Baseline aggregates outcomes {0, 5, 10, 15}, Risk increases aggregates {2, 14}, Risk decreases aggregates {8, 11}, Cost increases aggregates {1, 13}, Cost decreases aggregates {4, 7}, Neutral aggregates {3, 12}, and Mixed aggregates {6, 9}. Category

M=1

M=3

M=5

M = 10

M = 30

Baseline Risk increases Risk decreases Cost increases Cost decreases Neutral Mixed

0.789 0.024 0.003 0.038 0.122 0.012 0.013

0.862 0.012 0.002 0.034 0.088 0.001 0.001

0.887 0.009 0.002 0.026 0.076 0.000 0.000

0.916 0.006 0.001 0.016 0.061 0.000 0.000

0.943 0.004 0.000 0.007 0.047 0.000 0.000

46

Table 7: Audit outcomes and aggregated audit outcomes for a binary classification task on the Bank dataset using ensembles of independently fine-tuned BIG-SCIENCE T0 models. (a) Table reporting the fraction of test instances assigned to each outcome label o ∈ {0, . . . , 15} from Table 1. The baseline model is the ensemble of the Rashomon warehouse RT , with model error e(x; RT ), and audit decision r(x; RT ), while the candidate models are ensembles of q = {1, . . . , Q} random draws of the Rashomon retail set RM,q , with model error e(x; RM,q ) and audit decision r(x; RM,q ). The tuple (e(x; RT ), r(x; RT ), e(x; RM,q ), r(x; RM,q )) decides the combined audit outcomes, with label o ∈ {0, . . . , 15}. The table reports the mean ± standard deviation of the fraction of test instances assigned to each outcome label across q = {1, . . . , Q} random retail ensemble draws for different values of M . A desirable auditing strategy is one that minimizes excess audit costs and risks by increasing the Baseline category.

o

Meaning

M=1

M=3

M=5

M = 10

0

Baseline

0.562 ± 0.024

0.579 ± 0.009

0.583 ± 0.007

0.586 ± 0.003

1

Cost ↑

0.024 ± 0.021

0.009 ± 0.009

0.005 ± 0.007

0.002 ± 0.003

2

Risk ↑

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

3

Neutral

0.002 ± 0.003

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

4

Cost ↓

0.097 ± 0.029

0.081 ± 0.021

0.075 ± 0.018

0.071 ± 0.015

5

Baseline

0.089 ± 0.026

0.124 ± 0.021

0.135 ± 0.018

0.145 ± 0.015

6

Risk ↑, Cost ↓

0.005 ± 0.004

0.000 ± 0.001

0.000 ± 0.000

0.000 ± 0.000

7

Cost ↓

0.036 ± 0.010

0.022 ± 0.006

0.017 ± 0.005

0.011 ± 0.003

8

Risk ↓

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

9

Risk ↓, Cost ↑

0.000 ± 0.001

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

10

Baseline

0.050 ± 0.004

0.052 ± 0.002

0.053 ± 0.001

0.054 ± 0.001

11

Risk ↓

0.004 ± 0.004

0.002 ± 0.002

0.001 ± 0.001

0.000 ± 0.001

12

Neutral

0.004 ± 0.003

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

13

Cost ↑

0.026 ± 0.008

0.018 ± 0.006

0.015 ± 0.005

0.011 ± 0.004

14

Risk ↑

0.047 ± 0.015

0.034 ± 0.010

0.030 ± 0.009

0.028 ± 0.007

15

Baseline

0.055 ± 0.013

0.078 ± 0.010

0.086 ± 0.008

0.093 ± 0.007

(b) Summary of the fraction of test instances assigned to each high-level audit category obtained by aggregating the corresponding outcome labels from Table 7a. Specifically, Baseline aggregates outcomes {0, 5, 10, 15}, Risk increases aggregates {2, 14}, Risk decreases aggregates {8, 11}, Cost increases aggregates {1, 13}, Cost decreases aggregates {4, 7}, Neutral aggregates {3, 12}, and Mixed aggregates {6, 9}. Category

M=1

M=3

M=5

M = 10

Baseline Risk increases Risk decreases Cost increases Cost decreases Neutral Mixed

0.756 0.047 0.004 0.050 0.133 0.006 0.005

0.833 0.034 0.002 0.027 0.103 0.000 0.000

0.857 0.030 0.001 0.020 0.092 0.000 0.000

0.878 0.028 0.000 0.013 0.082 0.000 0.000

47

Table 8: Audit outcomes and aggregated audit outcomes for a binary classification task on the Adult dataset using ensembles of independently fine-tuned BIG-SCIENCE T0 models. (a) Table reporting the fraction of test instances assigned to each outcome label o ∈ {0, . . . , 15} from Table 1. The baseline model is the ensemble of the Rashomon warehouse RT , with model error e(x; RT ), and audit decision r(x; RT ), while the candidate models are ensembles of q = {1, . . . , Q} random draws of the Rashomon retail set RM,q , with model error e(x; RM,q ) and audit decision r(x; RM,q ). The tuple (e(x; RT ), r(x; RT ), e(x; RM,q ), r(x; RM,q )) decides the combined audit outcomes, with label o ∈ {0, . . . , 15}. The table reports the mean ± standard deviation of the fraction of test instances assigned to each outcome label across q = {1, . . . , Q} random retail ensemble draws for different values of M . A desirable auditing strategy is one that minimizes excess audit costs and risks by increasing the Baseline category.

o

Meaning

M=1

M=3

M=5

M = 10

0

Baseline

0.580 ± 0.011

0.590 ± 0.006

0.595 ± 0.004

0.598 ± 0.002

1

Cost ↑

0.019 ± 0.010

0.010 ± 0.005

0.005 ± 0.004

0.002 ± 0.002

2

Risk ↑

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

3

Neutral

0.001 ± 0.002

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

4

Cost ↓

0.082 ± 0.017

0.062 ± 0.012

0.058 ± 0.010

0.054 ± 0.008

5

Baseline

0.108 ± 0.019

0.144 ± 0.014

0.154 ± 0.012

0.163 ± 0.009

6

Risk ↑, Cost ↓

0.005 ± 0.004

0.000 ± 0.001

0.000 ± 0.000

0.000 ± 0.000

7

Cost ↓

0.035 ± 0.008

0.023 ± 0.005

0.018 ± 0.004

0.013 ± 0.003

8

Risk ↓

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

9

Risk ↓, Cost ↑

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

10

Baseline

0.046 ± 0.002

0.048 ± 0.001

0.048 ± 0.001

0.049 ± 0.001

11

Risk ↓

0.004 ± 0.002

0.002 ± 0.001

0.001 ± 0.001

0.001 ± 0.001

12

Neutral

0.004 ± 0.003

0.000 ± 0.000

0.000 ± 0.000

0.000 ± 0.000

13

Cost ↑

0.026 ± 0.005

0.020 ± 0.004

0.016 ± 0.004

0.012 ± 0.003

14

Risk ↑

0.035 ± 0.009

0.023 ± 0.006

0.020 ± 0.005

0.017 ± 0.004

15

Baseline

0.056 ± 0.010

0.078 ± 0.007

0.084 ± 0.007

0.091 ± 0.005

(b) Summary of the fraction of test instances assigned to each high-level audit category obtained by aggregating the corresponding outcome labels from Table 8a. Specifically, Baseline aggregates outcomes {0, 5, 10, 15}, Risk increases aggregates {2, 14}, Risk decreases aggregates {8, 11}, Cost increases aggregates {1, 13}, Cost decreases aggregates {4, 7}, Neutral aggregates {3, 12}, and Mixed aggregates {6, 9}. Category

M=1

M=3

M=5

M = 10

Baseline Risk increases Risk decreases Cost increases Cost decreases Neutral Mixed

0.790 0.035 0.004 0.045 0.117 0.005 0.005

0.860 0.023 0.002 0.030 0.085 0.000 0.000

0.881 0.020 0.001 0.021 0.076 0.000 0.000

0.901 0.017 0.001 0.014 0.067 0.000 0.000

48

Discrepancy Local discrepancy Prediction range Prediction variance

Figure 7: Scatter plot showing the relationship between our proposed consistency measure cσ at a test instance x (x-axis), and four different multiplicity metrics (rows) computed over the Rashomon warehouse, and defined in Section A.5, for different retail set sizes (columns). Each colored dot represents one of 200 randomly sampled test instances. Our proposed measure assigns higher scores to test instances with low multiplicity, and this trend becomes clearer as the ensemble size increases. Results are shown for paraphrase detection on the MRPC dataset using an ensemble of independently fine-tuned BERT models.

49

Discrepancy Local discrepancy Prediction range Prediction variance

Figure 8: Scatter plot showing the relationship between hcσ from Hamman et al. (2025), and four different multiplicity metrics (rows) computed over the Rashomon warehouse, and defined in Section A.5, for different retail set sizes (columns). Each colored dot represents one of 200 randomly sampled test instances. Unlike our proposed measure cσ , the hcσ measure assigns both high and low scores to test instances with low multiplicity, incorrectly identifying consistent test instances as inconsistent. Results are shown for paraphrase detection on the MRPC dataset using an ensemble of independently fine-tuned BERT models.

50

Table 9: Absolute Spearman correlation between consistency measures and multiplicity metrics when applied to the (9a) paraphrase detection task on the MRPC dataset, and (9b) sentiment classification task on the SST2 dataset using an ensemble of independently fine-tuned RoBERTa models. Higher values indicate greater agreement between the consistency measures (defined in Appendix A.6) computed for small retail ensembles and the multiplicity metrics (defined in Appendix A.5) computed from the entire Rashomon warehouse set. Rows highlighted in light blue correspond to the proposed consistency measures, cσ (x) and lmcσ (x). (a) MRPC

(b) SST2

Consistency measures Multiplicity metrics using using Rashomon warehouse retail pd(x) d(x) ldσ (x) pv(x) pr(x) ensembles M =1 cσ (x) 0.777 0.777 0.783 0.864 0.844 mg(x) 0.763 0.763 0.769 0.857 0.835 lmcσ (x) 0.765 0.765 0.777 0.849 0.829 lmgσ (x) 0.749 0.749 0.765 0.838 0.817 lvσ (x) 0.734 0.734 0.739 0.800 0.782 hcσ (x) 0.558 0.558 0.561 0.698 0.689 M =5 cσ (x) 0.875 0.875 0.888 0.952 0.927 mg(x) 0.873 0.873 0.883 0.951 0.924 lmcσ (x) 0.864 0.864 0.888 0.942 0.916 lmgσ (x) 0.860 0.860 0.888 0.938 0.911 lvσ (x) 0.808 0.808 0.829 0.889 0.872 hcσ (x) 0.582 0.582 0.595 0.733 0.723 M = 10 cσ (x) 0.891 0.891 0.896 0.968 0.944 mg(x) 0.887 0.887 0.887 0.966 0.941 lmcσ (x) 0.882 0.882 0.902 0.960 0.933 lmgσ (x) 0.877 0.877 0.899 0.956 0.928 lvσ (x) 0.858 0.858 0.878 0.937 0.918 hcσ (x) 0.588 0.588 0.599 0.743 0.734

Multiplicity metrics using Rashomon warehouse pd(x) d(x) ldσ (x) pv(x) pr(x) M =1 0.661 0.661 0.679 0.866 0.845 0.631 0.631 0.646 0.845 0.820 0.665 0.665 0.683 0.852 0.832 0.639 0.639 0.666 0.821 0.802 0.647 0.647 0.664 0.819 0.797 0.287 0.287 0.295 0.530 0.507 M =5 0.711 0.711 0.725 0.941 0.922 0.698 0.698 0.703 0.897 0.872 0.705 0.705 0.726 0.937 0.922 0.697 0.697 0.721 0.930 0.914 0.703 0.703 0.720 0.925 0.910 0.210 0.210 0.210 0.475 0.452 M = 10 0.730 0.730 0.749 0.962 0.943 0.727 0.727 0.734 0.929 0.908 0.725 0.725 0.750 0.960 0.944 0.711 0.711 0.738 0.947 0.932 0.723 0.723 0.747 0.959 0.943 0.284 0.284 0.301 0.563 0.541

51

Table 10: Absolute Spearman correlation between consistency measures and multiplicity metrics when applied to a binary classification task on the (10a) Bank dataset, and the (10b) Adult dataset using an ensemble of independently fine-tuned BIG-SCIENCE T0 models. Higher values indicate greater agreement between the consistency measures (defined in Appendix A.6) computed for small retail ensembles and the multiplicity metrics (defined in Appendix A.5) computed from the entire Rashomon warehouse set. Rows highlighted in light blue correspond to the proposed consistency measures, cσ (x) and lmcσ (x). (a) Bank

(b) Adult

Consistency measures Multiplicity metrics using using Rashomon warehouse retail pd(x) d(x) ldσ (x) pv(x) pr(x) ensembles M =1 cσ (x) 0.792 0.792 0.803 0.892 0.869 mg(x) 0.791 0.791 0.802 0.890 0.868 lmcσ (x) 0.795 0.795 0.807 0.895 0.873 lmgσ (x) 0.794 0.794 0.806 0.893 0.871 lvσ (x) 0.530 0.530 0.533 0.646 0.625 hcσ (x) 0.068 0.068 0.057 0.126 0.118 M =5 cσ (x) 0.840 0.840 0.857 0.940 0.919 mg(x) 0.840 0.840 0.856 0.939 0.918 lmcσ (x) 0.856 0.856 0.869 0.957 0.933 lmgσ (x) 0.856 0.856 0.869 0.956 0.932 lvσ (x) 0.631 0.631 0.647 0.763 0.745 hcσ (x) 0.061 0.061 0.048 0.120 0.112 M = 10 cσ (x) 0.859 0.859 0.875 0.950 0.932 mg(x) 0.858 0.858 0.875 0.949 0.932 lmcσ (x) 0.859 0.859 0.875 0.950 0.932 lmgσ (x) 0.858 0.858 0.875 0.948 0.931 lvσ (x) 0.666 0.666 0.681 0.783 0.771 hcσ (x) 0.058 0.058 0.046 0.116 0.110

Multiplicity metrics using Rashomon warehouse pd(x) d(x) ldσ (x) pv(x) pr(x) M =1 0.734 0.733 0.779 0.835 0.828 0.728 0.728 0.750 0.833 0.823 0.713 0.713 0.775 0.819 0.810 0.704 0.704 0.753 0.813 0.801 0.592 0.592 0.710 0.694 0.701 0.011 0.011 0.045 0.121 0.125 M =5 0.797 0.797 0.823 0.876 0.871 0.802 0.802 0.800 0.884 0.876 0.796 0.796 0.839 0.870 0.865 0.801 0.801 0.827 0.876 0.868 0.538 0.538 0.667 0.653 0.661 0.009 0.009 0.047 0.132 0.137 M = 10 0.865 0.865 0.885 0.946 0.940 0.869 0.869 0.868 0.950 0.941 0.859 0.859 0.895 0.940 0.932 0.864 0.864 0.886 0.944 0.933 0.628 0.628 0.744 0.737 0.743 0.005 0.005 0.050 0.135 0.137

52

Related documents

• Spatially Adaptive Ensemble Learning with Calibrated Predictive Uncertainty#997412
• A COST-AWARE FRAMEWORK FOR EXPLAINABILITY-BASED MONITORING OF BEHAVIORAL DRIFT IN CREDIT RISK MODELS#263293
• One Human, $N$ Agents: Audit-Budget Allocation for LLM Agent Fleets under Miscalibrated, Correlated Confidence#414160
• How to Verify Probabilistic Consistency of Predictive Models#973309
• How Spatial Uncertainty in Probabilistic Weather Forecasts Shapes Risk Perception#16498