Evaluating the Relevance of Uncertainty Estimators for LLM Hallucination
arXiv:2605.27016v1 [cs.CL] 26 May 2026
Yedidia AGNIMO∗ Ekimetrics Centre Inria de l’Université Grenoble Alpes France [email protected] Annabelle B LANGERO Ekimetrics France [email protected]
Anna KORBA CREST, ENSAE Institut Polytechnique de Paris France [email protected] Nicolas C HESNEAU Ekimetrics France [email protected]
Karteek A LAHARI Centre Inria de l’Université Grenoble Alpes France [email protected]
Abstract Large language models (LLMs) are prone to hallucinations, i.e., statements unsupported by the input or training data, hindering reliable deployment. In parallel, numerous uncertainty estimation (UE) methods have been proposed to quantify model confidence, and are often implicitly treated as proxies for model failure. However, the relationship between uncertainty and hallucinations remains insufficiently characterized. We present a systematic empirical study of the association between uncertainty estimators and hallucinations in LLMs. Rather than assuming this association, we evaluate directly when and to what extent it holds. We consider a diverse set of uncertainty metrics, including information-theoretic, samplingbased, and reflexive, and examine their behavior across hallucination settings. Our experiments cover both intrinsic hallucinations (violations of input faithfulness) and extrinsic hallucinations (unsupported claims relative to training data), using four complementary benchmarks, including RAGTruth and HalluLens . We find that the association is highly variable and often weak, depending on the hallucination type and the LLM under evaluation. These results challenge the use of uncertainty as a direct signal of hallucination and clarify when it provides actionable information.
1
Introduction
Large language models (LLMs) have demonstrated impressive performance across a wide range of natural language generation tasks, including question answering, summarization, and long-form reasoning [60]. However, despite their fluency and apparent coherence, LLMs often generate hallucinations, i.e., statements that are not supported by the evidence the model is supposed to rely on [4, 16]. Such unsupported generations pose a major obstacle to the safe and reliable deployment of LLMs, particularly in settings where errors can have serious consequences, such as in healthcare, law, and finance [5, 41]. ∗ Correspondence to [email protected].
Preprint.
Recent work has argued that hallucination should be distinguished from factuality, which refers to correctness relative to real-world truth, to better tackle each problem [2, 4]. They further distinguish intrinsic from extrinsic hallucinations, depending on the source of evidence used to assess groundedness. Intrinsic hallucinations occur when model responses are not supported by the information explicitly provided at inference time, whereas extrinsic hallucinations arise when responses are not supported by the knowledge the model is expected to have encountered in its pretraining data [4]. This distinction matters because the mechanisms underlying these two types, and therefore the signals available for detecting them, may differ substantially. At the same time, substantial effort has been devoted to developing uncertainty estimation (UE) methods for language models, which aim to quantify a model’s confidence in its responses [26, 47]. This includes information-theoretic measures based on token probabilities or likelihoods [18, 34], sampling and ensemble-based approaches that measure variability across generations [27, 31], and reflexive approaches that rely on model self-evaluation [25, 44]. In practice, such signals are often used for abstention or selective prediction and routing uncertain cases to additional verification [1, 55], model self-refinement [56], in-context learning [61], or adaptive retrieval [35]. However, uncertainty estimates are not designed specifically to detect hallucinations, and existing evaluations rarely test them against hallucination-specific targets. Most prior work assesses uncertainty with respect to broader notions of error, correctness, robustness, or calibration rather than with respect to explicitly defined hallucination criteria [3, 47, 51]. Moreover, many uncertainty scores used in practice are heuristic proxies without a rigorous probabilistic foundation. In this paper, we evaluate whether uncertainty estimators can support hallucination detection when hallucination is defined as inconsistency with expected evidence. We compare 46 estimators across 4 datasets and 3 open-weight instruction-tuned models: Mistral-7B-Instruct, Llama-2-7B-Chat, and Llama-2-13B-Chat. We will often refer to these datasets as tasks, as they cover different hallucination and task type: context faithfulness in retrieval-augmented generation (intrinsic) and extrinsic settings, including short-form question answering, long-form generation, and abstention from nonexistent entities. Our main contributions and findings are: • A hallucination-specific benchmark for uncertainty. We evaluate uncertainty estimators against explicit hallucination targets, distinguishing intrinsic context-faithfulness from extrinsic pretraining-data groundedness. • The relationship between uncertainty and hallucination depends on the task. Uncertainty estimators can help separate hallucinated from grounded responses, but their discriminative power varies substantially across datasets. In short-form question answering, most estimators provide strong discrimination. In long-form generation, AUROC values are more uniformly moderate, with smaller differences between estimators. For contextfaithfulness and abstention, the discrimination is also moderate, but it is concentrated in fewer estimators and varies more across models. Overall, these results show that the relationship between uncertainty and hallucination depends on the dataset, which encodes not only the hallucination type, but also the task type, response length, evidence source, and expected model behavior. This argues against treating uncertainty scores as general-purpose hallucination detectors. • No uncertainty estimator is consistently best across datasets. The estimators that perform best on one dataset do not reliably remain best on another. Rankings are more stable across models within the same dataset than across datasets for the same model, indicating that dataset variation affects both estimator choice and overall discriminative power. This argues against selecting a hallucination detector from a single aggregate ranking and instead supports dataset-specific validation of uncertainty estimators. • Competitive estimators are partially correlated. Although no uniformly best estimator emerges, the estimators that perform well tend to produce correlated response rankings across datasets and models. As a practical implication, selecting a few representatives from distinct groups of correlated estimators may be preferable to relying on a single estimator or using all available scores. 2
2
Background on uncertainty estimators
Notation. We consider an autoregressive language model pθ (y | x), where x denotes the input (query and optional context), y = (y1 , . . . , yT ) is a generated response sequence, and θ parametrizes the model. At each step t, the model produces a token-level probability distribution pθ (· | x, y<t ), so QT that pθ (y|x) = t=1 pθ (yt |x, y<t ) for an response sequence of length T , with y<t = (y1 , . . . , yt−1 ). 2.1
Uncertainty definition and scope
There is no universally accepted formal definition of uncertainty in language generation [3, 59]. In this work, uncertainty is treated as a query-level property of the predictive distribution pθ (· | x). A generic query-level target is denoted U ⋆ (x) = Φ(pθ (· | x)), where Φ is a property of the predictive distribution [8]. Different estimators may approximate different aspects of this target and need not share the same probabilistic foundation. Terminology in the literature is not fully consistent: some works use uncertainty and confidence interchangeably [39, 59], while others reserve confidence for response-level uncertainty y [31]. Here, uncertainty refers by default to query-level uncertainty. Scores of the form U (x) target this quantity directly, whereas response-conditioned scores U (x, y) are treated as single-response proxies rather than as a separate response-level notion. 2.2
Uncertainty estimators studied
In this work, LLM uncertainty estimators are first organized by access regime, separating white-box [WB] from black-box [BB] methods. Within these settings, the families correspond to groups of estimators with similar input requirements for uncertainty computation. The implementation follows LM-P OLYGRAPH [15, 47]; full estimator definitions are given in Section B. [WB] Information/logit-based methods. Logit- or information-based estimators derive uncertainty directly from the token-level predictive distributions pθ (· | x, y<t ) produced during autoregressive generation. They differ in how they aggregate token-level signals over a generated sequence. The simplest methods rely only on the probability of the generated token, yielding sequence-level scores such as negative log-likelihood and related confidence scores based on sequence probability, such as Maximum Sequence Probability (MSP), 1 − pθ (y | x) [18, 34]. Because MSP only uses the probability assigned to the observed sequence, entropy-based methods instead exploit the full token distribution, for example through mean token entropy [18] or similarity-to-uniform measures such as Rényi or Fisher–Rao divergences [11]. Pointwise mutual information methods compare conditional and unconditional token probabilities to quantify the extent to which the query shapes the generated response [46]. More recent variants incorporate semantic information into token probabilities. TokenSAR [13] upweights improbable tokens that are semantically important, while Claim-Conditioned Probability (CCP) focuses uncertainty estimation on token alternatives that preserve or alter the meaning of the current claim [14]. These methods are efficient because they only require a single forward pass to obtain logits, and provide the most direct readout of the model’s predictive distribution. [WB] Sample-based methods. White-box sample-based methods extend single-pass estimators by repeatedly sampling responses for the same query. Some use these samples to approximate sequence-level uncertainty quantities derived from the model’s predictive distribution, for example through Monte Carlo estimates of entropy or related likelihood-based functionals [34]. However, such estimates do not distinguish semantic variation from surface-form variation across samples. Semantic Entropy, for instance, addresses this limitation by clustering sampled responses into semantic equivalence classes and computing entropy over these discrete classes [27], whereas Semantic Density [39] relies on fine-grained, continuous pairwise semantic similarities instead of a binary semantic-equivalence partition. SentenceSAR [13] further replaces hard semantic clustering with soft semantic weighting, reweighting token-level negative log-likelihood contributions using semantic similarity across sampled responses, so that improbable and semantically isolated responses contribute more strongly to the uncertainty score. Cocoa [48] estimators instead multiply a token logitbased uncertainty score by a semantic-variability score computed from the sampled responses. These methods capture predictive instability beyond a single forward pass but are more computationally expensive and may still fail when the model consistently reproduces the same hallucination. 3
[WB] Internal-state methods. Internal-state methods exploit signals encoded in the model’s latent computations, using attention patterns or hidden representations as indicators of predictive unreliability. Some measure local processing signals, like AttentionScore that quantify the extent to which a token attends to itself across heads and layers, [43], while others model sequential propagation effects, as in RAUQ [50], or reweight token probabilities using attention-derived saliency [32]. The common premise is that uncertainty may already be reflected in the geometry or dynamics of the forward pass before it becomes visible in the final response distribution. [WB] Training-based methods. Density- or training-based estimators quantify uncertainty through similarity to the training distribution. They approximate the distribution of training representations, typically using Gaussian densities, and estimate how far an input or representation lies from highdensity training regions, often through Mahalanobis distance [29, 49]. These methods are especially relevant for out-of-distribution (OOD) detection and are computationally efficient at inference time, but require access to training data. [WB] Reflexive methods. Reflexive methods estimate uncertainty by querying the model about its own confidence. Rather than deriving uncertainty from predictive distributions or internal representations, they rely on the model’s explicit self-assessment of the reliability of its answers. One representative example is P (True) prompting, which uses the probability assigned to the token true after a follow-up prompt such as “Are you sure?” [25]. These methods are flexible and straightforward to implement, but their reliability depends on the model’s ability to express uncertainty accurately and in a calibrated manner, as well as on the prompt formulation. [BB] Black-box estimators. Finally, in many practical settings, only generated text is accessible. Black-box methods therefore estimate uncertainty from responses alone, typically by sampling multiple responses, constructing a pairwise lexical or semantic similarity matrix between them, and summarizing its structure into an uncertainty score [31, 36]. While broadly deployable, they are often computationally expensive.
3
Benchmark setup
3.1
Datasets
The evaluation suite spans two types of hallucinations. Intrinsic hallucination is evaluated with RAGTruth [37], where responses are judged against evidence provided at inference time. Extrinsic hallucination is evaluated with three HalluLens datasets [4]: PreciseWikiQA , LongWiki , and NonExistentRefusal , where responses are judged against knowledge expected to be available from pretraining data. Each dataset (task) is associated with a response-level quality metric Q(x, y), derived from its (specific) definition of hallucination. Q serves as the target variable against which uncertainty estimators are assessed: we test whether higher uncertainty is associated with lower response quality. Intrinsic hallucination: RAGTruth . RAGTruth is a human-annotated dataset for hallucination detection in retrieval-augmented generation [37], spanning summarization, passage-grounded question answering, and data-to-text generation from structured business data. Each instance contains a query, retrieved context, model response, and span-level annotations that mark content that is unsupported by or conflicts with the provided context. We use the processed Hugging Face release2 , which standardizes metadata and annotations. Under this context-faithfulness definition, information not supported by the retrieved context is treated as hallucinated, even if it is factually correct according to external knowledge. We derive a response-level target from the span annotations: hj = 1 if response yj contains at least one hallucinated span, and hj = 0 otherwise, with quality target Q(xj , yj ) = 1 − hj . Extrinsic hallucination: HalluLens . The three HalluLens datasets evaluate complementary settings of extrinsic hallucination, defined as inconsistency with information expected to be available in the model’s pretraining data [4]. Rather than measuring only generic answer correctness, HalluLens probes deviations from pretraining knowledge: Wikipedia serves as a proxy for broadly available pretraining data, while synthetic nonexistent entities test behavior on information expected to lie outside it. The datasets cover distinct response formats: PreciseWikiQA evaluates short answers, LongWiki evaluates long-form generation, and NonExistentRefusal evaluates whether the model abstains when queried about nonexistent entities. We follow the HalluLens procedure [4]. For 2 https://huggingface.co/datasets/wandb/RAGTruth-processed
4
Dataset P RECISE W IKI QA L ONG W IKI N ON E XISTENT R EFUSAL RAGT RUTH
Train + eval queries
Avg. document length
Avg. target length
1000 + 2000 1000 + 2000 1000 + 2000 1000 + 1960
37.9 38.4 17.6 698.2
5.8 235.0 – 194.4∗
Table 1: Summary statistics of the benchmark datasets. Token counts with Llama-2’s tokenizer. ∗ For RAGT RUTH, the reported target length corresponds to the average generated response length. N ON E XISTENT R EFUSAL has no reference target answer because the desired behavior is abstention. PreciseWikiQA and LongWiki , examples are built from sampled Wikipedia pages: a question and a gold answer are generated from the page. The gold answer is the response to the question, which serves as a reference for evaluation. The evaluated models then produce responses to these questions. Because the datasets do not provide direct human hallucination annotations, the same LLM3 is used as judge: for PreciseWikiQA and LongWiki , it evaluates model responses against the gold answers; for NonExistentRefusal , it evaluates whether the response is a correct abstention. We therefore treat the three datasets as separate evaluation tasks, each with its own response-level target (refer to Bang et al. [4] for more details). For PreciseWikiQA , Q(xj , yj ) is binary answer correctness relative to the gold answer. For LongWiki , the judge decomposes both the gold answer and the model response into atomic claims. Precision is the proportion of response claims supported by the gold answer, and recall is the proportion of gold-answer claims recovered by the response; quality is their harmonic mean. Following Bang et al. [4], claim counts are capped at K = 32, yielding the Q(xj , yj ) = F1@K(xi , yj ) target. 3.2
Choice of models and computation of uncertainty estimators
Models. The model selection is constrained by RAGTruth , where each query is associated with six model-generated responses annotated by humans for hallucination [37]. Among these models, the three open-weight models are retained: Mistral-7B-Instruct-v0.2, Llama-2-7b-chat-hf, and Llama-2-13b-chat-hf. This choice ensures token-logit access and improves reproducibility under a controlled evaluation protocol. Recovering token logits for annotated RAGTruth responses. Because RAGTruth provides responses generated during dataset construction and later annotated for hallucinations, evaluation must be performed on these annotated responses. Regenerating responses would break the correspondence with the original annotations. However, token logits are not provided and must be recovered for information-based and white-box sample-based estimators, which require token-level probabilities on the provided generation. In our study, we recover these probabilities by scoring each annotated response under the corresponding causal language model with teacher forcing [53]. Given an input–response pair (x, y), we run the model on the concatenated sequence x ⊕ y and extract logits at the response positions. This yields the autoregressive conditionals pθ (yt | x, y<t ), i.e., the probability assigned to each observed token given the input and observed prefix. These probabilities are then used by uncertainty estimators requiring token-level logits for the annotated generation. Dataset splits for training-based uncertainty estimators. Training-based estimators require training data. Following Vashurin et al. [47], RAGTruth is partitioned at the query level into 1000 training and 2000 evaluation queries, ensuring that all responses associated with a query remain in the same split. For HalluLens , 3000 question-answer pairs are constructed following Bang et al. [4] and split using the same 1000/2000 train-evaluation ratio. All uncertainty estimators are evaluated on the same 2000 evaluation instances; the 1000-instance training split is used only by training-based estimators requiring fitting or training-set statistics. Summary statistics are reported in Table 1. Auxiliary reference-based score: AlignScore. We also report AlignScore [58] as an auxiliary comparison score. It is reference-based because it evaluates a response by comparing it to an external reference text, such as a gold answer, a source Wikipedia page, or an abstention template. AlignScore is not an uncertainty estimator but an encoder-based model fine-tuned to estimate whether 3 hugging-quants/Meta-Llama-3.1-70B-Instruct-AWQ-INT4
5
one text is supported by another. For PreciseWikiQA and LongWiki , AlignScore(y, ref) measures whether claims in the model response are supported by the gold answer or source Wikipedia page, yielding a precision-like support signal. The reverse score, AlignScore(ref, y), measures whether claims in the reference are recovered by the response, yielding a recall-like recovery signal. For NonExistentRefusal , where no gold answer is available, responses are compared against canonical abstention templates (Section C.1). We use AlignScore to assess the strength of the association between these reference-based signals and the response quality measured by Q. This comparison also helps indicate whether a dataset target is closer to a precision-like support measure, a recalllike recovery measure, or a mixture of both, while making explicit that AlignScore uses reference information unavailable to uncertainty estimators.
4
Evaluation methodology
Let D = {(xj , yj )}nj=1 denote an evaluation set, where xj is the input and yj is the model response. Each dataset defines a response-level quality target {Q(xj , yj )}ni=1 , as described in Section 3.1. Each uncertainty estimator produces a scalar score uj for the same instance. Our evaluation assesses whether larger uncertainty values are associated with lower hallucination-specific quality. Uncertainty scores are computed with LM-P OLYGRAPH [47] and evaluated along four axes: discrimination, selective prediction, rank calibration, and redundancy. All reported metrics are accompanied by bootstrap estimates of statistical variability. For each dataset-model pair, 1,000 bootstrap replicates are generated from the evaluation set, each metric is computed on every replicate, and the resulting standard deviation is reported. 4.1
Evaluating hallucination detection
Discriminating hallucinated from supported responses. The first evaluation axis tests whether uncertainty separates lower-quality or hallucinated generations from higher-quality ones. For datasets with binary response-level targets (RAGTruth , PreciseWikiQA , NonExistentRefusal ), AUROC is reported with lower-quality or hallucinated responses as the positive class. For LongWiki , the claimbased F1@K target is binarized at 0.5 to enable the same AUROC comparison. This binarization is restricted to discrimination analysis; continuous F1@K is retained for selective-prediction analyses. Selective prediction. A complementary axis evaluates whether uncertainty can support rejectionbased quality control. In selective prediction, high-uncertainty instances are removed, and the quality of the retained responses is measured as the rejection rate increases. Let π a permutation of {1, . . . , n} denote the ordering induced by the uncertainty scores {uj }ni=1 , sorted from lowest to highest uncertainty. For a rejection rate r ∈ [0, 1], the retained subset is S(r) = {π1 , . . . , π⌊(1−r)n⌋ }. P 1 The average retained quality is Q(r) = |S(r)| j∈S(r) Q(xj , yj ), which defines a quality-retention curve r 7→ Q(r). The oracle orders instances by decreasing quality, equivalently rejecting the lowest-quality ones first. Following prior works [34, 47], the curve is summarized with the Prediction Rejection Ratio (PRR), AUCU − AUCrandom , PRR = AUCoracle − AUCrandom where AUCU is the area under the quality–retention curve induced by the uncertainty estimator, AUCrandom corresponds to random rejection, and AUCoracle corresponds to the optimal ordering induced by the quality target. Higher PRR values indicate that rejecting high-uncertainty instances yields larger quality gains relative to random rejection. Rank calibration. Discrimination and selective prediction assess separability and rejection utility; rank calibration instead asks whether the uncertainty ordering remains globally aligned with expected quality. Rank-Calibration Error (RCE; 21) is used as a rank-based diagnostic of whether uncertainty scores are monotonically aligned with response quality. RCE allows comparing estimators with different scales, including unbounded, and does not require ad hoc binarization of continuous targets. 4.2
Redundancy and complementarity across estimators
After evaluating individual performance, we assess whether estimators capture distinct or overlapping signals. Redundancy is analyzed at two levels: similarity in instance rankings within each dataset6
model pair (which we refer to as a panel), and similarity in performance profiles across datasets and models. Together, these analyses indicate whether uncertainty estimator groups are interchangeable or complementary, and whether heterogeneity is driven primarily by tasks or models. Estimator-level correlation. For each panel (dataset-model pair), we compute pairwise Spearman correlations between estimator scores to measure how similarly two estimators rank instances. The resulting correlation matrices are aggregated and visualized with hierarchical clustering to identify groups of estimators with similar behavior. Performance-level correlation. Score-level similarity within a panel does not show whether estimators succeed in alternative evaluation settings. We therefore compare estimator performance profiles across panels. For each estimator, AUROC and PRR values are collected across all dataset-model pairs, and Kendall’s τ is computed between these profiles to assess whether the estimators exhibit similar performance patterns across tasks and models.
5
Results
We report results on four datasets covering different hallucination types, RAGTruth (RT), PreciseWikiQA (PWQA), LongWiki (LW), and NonExistentRefusal (NR), and three models: Mistral-7B-Instruct-v0.2 (M7B), Llama-2-7B-chat (L7B), and Llama-2-13B-chat (L13B). This yields 12 dataset-model pairs, referred to as panels. Unless stated otherwise, estimator rankings exclude AlignScore variants, which are used as reference-based comparators. AUROC, prediction–rejection ratio (PRR), and rank-calibration error (RCE) induce similar estimator rankings across panels (median Spearman correlation +0.97 for AUROC–PRR; −0.96 for AUROC–RCE, where the negative sign reflects that lower RCE is better; Table 8). We therefore use AUROC as the primary discrimination metric, and refer to PRR and RCE only when they add distinct information. Our empirical results, detailed below, provide insights on the relevance of uncertainty estimators for LLM hallucination by assessing their discriminative power, the heterogeneity of results across hallucination types and whether the best estimators provide complementary or redundant signal. Average performance and consistency across panels. Figure 1 compares each estimator’s mean AUROC across panels with the variability of its within-panel rank. Estimators in the upper part of the plot discriminate hallucinations more strongly on average; estimators on the left rank more consistently across panels. CCP (logit/information-based) and CocoaMSP (white-box sample-based) provide the best compromise between the two criteria. MSP (logit/information-based) and AttentionScore (internal-state) achieve comparable or higher mean AUROC but exhibit greater rank variability, indicating that their averages are driven by panels where they peak rather than by uniform performance across panels. Training-based density estimators (Mahalanobis Distance, MD; Relative MD; Robust Density Estimation, RDE) and the information-based pointwise mutual information variants (PMI, CPMI) recur near the bottom of the plot. The pooled view, therefore, gives the main warning: uncertainty is informative, but no estimator is uniformly reliable. In the following, we show that the variability in Figure 1 is driven primarily by the dataset and secondarily by the model. The dataset is the primary source of heterogeneity in estimator rankings. Estimator rankings vary more with dataset changes than with model changes. This pattern is reasonable in hindsight, and thus reassuring, but it was not trivial a priori. The Kendall’s τ heatmaps in Figures 5 and 6 (see appendix) quantify this pattern: when the model is fixed and the dataset changes, mean estimator-ranking agreement is low (τ = 0.10 to 0.15); when the dataset is fixed and the model changes, agreement is higher but uneven, with stronger transfer on LW (τ = 0.71) and PWQA (τ = 0.62) than on RT (τ = 0.35) and NR (τ = 0.30). We investigate this heterogeneity further by comparing the behavior of the estimators across hallucination types. Figure 2 reports family-level predictive performance of uncertainty estimators for two contrasting hallucination settings: PWQA, where most estimators exhibit moderate to strong discriminative power, and RT, where it is sparsest (see Section 2.2 for the family definitions). On PWQA, all six uncertainty estimator families lie above the 0.5 baseline of an uninformative score, with the information-based and white-box sample-based families reaching the highest family-aggregate AUROCs. Hence, our results indicate that uncertainty estimators provide useful discriminative power for hallucination on PWQA. In contrast, family-level ROC curves on RT sit closer to the 0.5 baseline, suggesting that uncertainty estimators carry limited information for discriminating hallucinations in this setting. The two remaining datasets, LW and NR, are reported in Figure 7 (see appendix). 7
AS-prec-gold
Mean AUROC (↑ better)
0.70
CCP AS-recall-gold MSP CocoaMSP
0.65
sentSAR
0.60
AttentionScore
0.55 BB PTrue
0.50 0.45
CPMI
PMI
0.40
Information-based Sampling-based Internal State Training-based Reflexive Black-box AlignScore
RDE RMD MD
6
8
10
12
14
16
18
Rank std across panels (↓ more consistent)
Figure 1: Mean AUROC against rank variability across the 12 panels. Each point represents one uncertainty estimator or AlignScore variant. Rank variability is the standard deviation of an estimator’s within-panel rank across panels; lower values indicate a more stable cross-panel ranking. AlignScore variants are reference-based comparators.
Model differences still affect which estimators perform best. Although task differences dominate overall, the model still changes which uncertainty signal is most informative on a given dataset. On RT with M7B, the leading estimators include the information-based MSP and CCP, as well as the white-box, sample-based CocoaMSP (see Table 4). This pattern does not persist on the Llama models. On RT with L13B, by contrast, reflexive estimators, in which the language model is prompted to assess its own uncertainty (e.g., PTrue), perform best, whereas they are uninformative on the other models. Thus, even within a fixed task setting, estimator choice remains model-dependent. RAGTruth
True Positive Rate
1.0
PreciseWiki
0.8 0.6 0.486 0.464 0.489 0.351 0.527 0.423 0.585
0.4 0.2 0.0
0.0
0.2
0.4
0.6
0.8
1.0 0.0
0.586 0.702 0.628 0.558 0.503 0.702 0.762
0.2
False Positive Rate Information-based Sampling-based
0.4
0.6
0.8
1.0
False Positive Rate Internal State Training-based
Reflexive Black-box
AlignScore
Figure 2: Family-level ROC aggregates per task, averaged across the three generators. For each task–model panel, estimator ROC curves are first averaged within each family by taking the mean true positive rate at each false positive rate. For each task, these model-level family means are then averaged across models. Shaded bands show one standard deviation of these model-level family-mean true positive rates at each false positive rate; they reflect cross-model variability. Some families consistently perform poorly. The bottom of the panel ranking in Figure 1 is more stable than the top across datasets and models (see Figure 4 in the appendix). The training-based density estimators (MD, Relative MD, RDE) are consistently among the bottom five in nearly every panel, with the information-based PMI and CPMI variants following closely. A plausible interpretation is that density-based estimators measure how typical a generated response is under a fitted reference representation distribution. Since hallucinated responses are still sampled from the generator, they need not be atypical in representation space. Performance may also depend on modeling choices such as the training data, background corpus, representation layer and dimensionality reduction. These results suggest that training-based density estimators and PMI/CPMI should not be used as default uncertainty signals without dataset-specific validation. 8
e
c
c-
Ec
c-
Ec
co aM SP
Co
P
SP
M
R
CC
SA
A
tte nt
io
nS co re
Spearman ρ (12 panels)
The recurrent top estimators form three correlated ranking clusters. Figure 3 reports Spearman’s ρ correlations between the 7 estimators with the highest mean AUROC after averaging across panels. This figure shows that three correlated ranking clusters emerge. The first cluster is logit- and sample-based, and relies on logits: SAR, Maximum Sequence Probability (MSP), Claim-Conditioned Probability (CCP), and CocoaMSP. The strongest correlations occur between MSP, CCP, and CocoaMSP (ρ ≥ 0.78), while SAR is more weakly attached to the same cluster. This correlation is informative because these estimators differ in construction: MSP is a direct sequence-likelihood score; CCP filters local token alternatives through a natural-language inference model; CocoaMSP modulates an MSP-derived score using inter-sample semantic dissimilarity; and SAR combines token-level and sentence-level relevance signals across sampled responses. The second cluster contains the two Eccentricity variants, based respectively on contradic- AttentionScore 1.00 tion (Ecc-c) and entailment (Ecc-e) relations. 0.75 SAR 0.06 Both are black-box graph-based estimators: they 0.50 CCP 0.13 0.35 build a semantic-relation graph over sampled 0.25 responses and quantify dispersion from the reMSP 0.13 0.47 0.82 0.00 sulting graph spectral representation. Their −0.25 moderate correlation (ρ = 0.36) suggests that CocoaMSP 0.06 0.41 0.78 0.90 −0.50 entailment- and contradiction-based graph conEcc-c -0.09 0.06 0.17 0.06 0.16 structions are related but not interchangeable. −0.75 The third cluster is a singleton, AttentionScore, −1.00 Ecc-e -0.05 0.30 0.19 0.22 0.29 0.36 which uses internal-state representations rather than logits or generated samples. This likely explains why its correlations with the other estimators remain low. Figure 3: Spearman correlation between the top uncertainty estimators across panels. Access constraints should shape estimator selection. The ranking clusters differ not only in performance, but also in input requirements. The first cluster mostly contains white-box estimators that require token probabilities. CocoaMSP and SAR further require sampled generations and a sentence-similarity model, while AttentionScore requires access to attention weights. In contrast, the NLI-graph estimators are black-box with respect to the generator: they use sampled text responses and an auxiliary NLI model, but do not require generator logits, hidden states, or attention weights. When white-box access is available, CocoaMSP and CCP perform better across datasets, while MSP remains an attractive, computationally efficient baseline.
6
Conclusion
Our study clarifies when existing uncertainty estimation methods align with hallucination-related output quality in large language models. The main finding is that uncertainty is informative, but only conditionally. Task-specific evaluation targets are the main source of heterogeneity, which argues against treating any uncertainty estimator as a universal hallucination detector. Instead, uncertainty estimators should be selected for practical use based on the target task, the level of access to the generator, the computational budget, and validation performance on the target setting. The benchmark also suggests a practical structure for estimator selection: (i) black-box Eccentricity variants are strong candidates when logits are unavailable, (ii) logit-based estimators such as MSP, CCP, and related methods provide a strong default family when white-box access is available, and (iii) AttentionScore appears most useful in context-faithfulness and abstention tasks.
Limitations and future work There are also a few limitations to our study. There are also a few limitations to our study. Each dataset determines whether a response is hallucinated according to its own annotation or reference protocol. In particular, LongWiki uses an F1 target that is a long-form quality signal rather than a pure hallucination label. Class imbalance, that is, variation across models in the proportion of responses labeled as hallucinated, may also affect panel-level comparisons, although bootstrap intervals partly quantify sampling variability. Finally, the evaluated model set is controlled and reproducible but does 9
not cover larger frontier systems or the full range of post-training regimes. The influence of other factors, such as the different phases of the training paradigm, could also be worth investigating. Future work could extend this evaluation in three directions. First, hallucination targets should be made more fine-grained and stratified. Second, localization-aware analyses should use span-level annotations, such as those available in RAGTruth, to test whether uncertainty scores indicate where hallucinations occur in the answer, rather than only whether a response is unreliable.
References [1] Abbasi-Yadkori, Y., Kuzborskij, I., György, A., and Szepesvari, C. (2024). To Believe or Not to Believe Your LLM: Iterative Prompting for Estimating Epistemic Uncertainty. In NeurIPS. [2] Augenstein, I., Baldwin, T., Cha, M., Chakraborty, T., Ciampaglia, G. L., Corney, D., DiResta, R., Ferrara, E., Hale, S., Halevy, A., Hovy, E., Ji, H., Menczer, F., Miguez, R., Nakov, P., Scheufele, D., Sharma, S., and Zagni, G. (2024). Factuality challenges in the era of large language models and opportunities for fact-checking. Nature Machine Intelligence, 6(8):852–863. [3] Bakman, Y. F., Yaldiz, D. N., Kang, S., Zhang, T., Buyukates, B., Avestimehr, S., and Karimireddy, S. P. (2025). Reconsidering LLM uncertainty estimation methods in the wild. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 29531–29556. [4] Bang, Y., Ji, Z., Schelten, A., Hartshorn, A., Fowler, T., Zhang, C., Cancedda, N., and Fung, P. (2025). HalluLens: LLM hallucination benchmark. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 24128–24156. [5] Bengio, Y., Mindermann, S., Privitera, D., Besiroglu, T., Bommasani, R., Casper, S., Choi, Y., Fox, P., Garfinkel, B., Goldfarb, D., Heidari, H., Ho, A., Kapoor, S., Khalatbari, L., Longpre, S., Manning, S., Mavroudis, V., Mazeika, M., Michael, J., Newman, J., Ng, K. Y., Okolo, C. T., Raji, D., Sastry, G., Seger, E., Skeadas, T., South, T., Strubell, E., Tramèr, F., Velasco, L., Wheeler, N., Acemoglu, D., Adekanmbi, O., Dalrymple, D., Dietterich, T. G., Felten, E. W., Fung, P., Gourinchas, P.-O., Heintz, F., Hinton, G., Jennings, N., Krause, A., Leavy, S., Liang, P., Ludermir, T., Marda, V., Margetts, H., McDermid, J., Munga, J., Narayanan, A., Nelson, A., Neppel, C., Oh, A., Ramchurn, G., Russell, S., Schaake, M., Schölkopf, B., Song, D., Soto, A., Tiedrich, L., Varoquaux, G., Yao, A., Zhang, Y.-Q., Albalawi, F., Alserkal, M., Ajala, O., Avrin, G., Busch, C., Carvalho, A. C. P. d. L. F. d., Fox, B., Gill, A. S., Hatip, A. H., Heikkilä, J., Jolly, G., Katzir, Z., Kitano, H., Krüger, A., Johnson, C., Khan, S. M., Lee, K. M., Ligot, D. V., Molchanovskyi, O., Monti, A., Mwamanzi, N., Nemer, M., Oliver, N., Portillo, J. R. L., Ravindran, B., Rivera, R. P., Riza, H., Rugege, C., Seoighe, C., Sheehan, J., Sheikh, H., Wong, D., and Zeng, Y. (2025). International AI Safety Report. arXiv:2501.17805 [cs]. [6] Bouchard, D., Chauhan, M. S., Skarbrevik, D., Ra, H.-K., Bajaj, V., and Ahmad, Z. (2026). UQLM: A Python Package for Uncertainty Quantification in Large Language Models. Journal of Machine Learning Research, 27(13):1–10. [7] Chen, C., Liu, K., Chen, Z., Gu, Y., Wu, Y., Tao, M., Fu, Z., and Ye, J. (2023). INSIDE: LLMs’ Internal States Retain the Power of Hallucination Detection. In International Conference on Learning Representations. [8] Chen, L., Melo, G. d., Suchanek, F. M., and Varoquaux, G. (2026). Query-Level Uncertainty in Large Language Models. In ICLR. [9] Cover, T. M. and Thomas, J. A. (2001). Elements of information theory. Wiley-Interscience, Hoboken, NJ. [10] Cruz, A. F., Hardt, M., and Mendler-Dünner, C. (2024). Evaluating language models as risk scores. In Proceedings of the 38th International Conference on Neural Information Processing Systems, volume 37, pages 97378–97407. [11] Darrin, M., Piantanida, P., and Colombo, P. (2023). Rainproof: An umbrella to shield text generator from out-of-distribution data. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 5831–5857. 10
[12] Devic, S., Srinivasan, T., Thomason, J., Neiswanger, W., and Sharan, V. (2025). From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centered. arXiv:2506.07461 [cs] version: 1. [13] Duan, J., Cheng, H., Wang, S., Zavalny, A., Wang, C., Xu, R., Kailkhura, B., and Xu, K. (2024). Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of FreeForm Large Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5050–5063. Association for Computational Linguistics. [14] Fadeeva, E., Rubashevskii, A., Shelmanov, A., Petrakov, S., Li, H., Mubarak, H., Tsymbalov, E., Kuzmin, G., Panchenko, A., Baldwin, T., Nakov, P., and Panov, M. (2024). Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantification. In Findings of the Association for Computational Linguistics, pages 9367–9385. [15] Fadeeva, E., Vashurin, R., Tsvigun, A., Vazhentsev, A., Petrakov, S., Fedyanin, K., Vasilev, D., Goncharova, E., Panchenko, A., Panov, M., et al. (2023). Lm-polygraph: Uncertainty estimation for language models. Proceedings of the Conference on Empirical Methods in Natural Language Processing: System Demonstrations. [16] Fan, D., Delsad, S., Flammarion, N., and Andriushchenko, M. (2026). HalluHard: A Hard Multi-Turn Hallucination Benchmark. arXiv:2602.01031 [cs]. [17] Farquhar, S., Kossen, J., Kuhn, L., and Gal, Y. (2024). Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625–630. [18] Fomicheva, M., Sun, S., Yankovskaya, L., Blain, F., Guzmán, F., Fishel, M., Aletras, N., Chaudhary, V., and Specia, L. (2020). Unsupervised quality estimation for neural machine translation. Transactions of the Association for Computational Linguistics, 8:539–555. [19] He, P., Liu, X., Gao, J., and Chen, W. (2021). DeBERTa: Decoding-enhanced BERT with disentangled attention. In International Conference on Learning Representations. [20] Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., and Liu, T. (2024a). A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. ACM Transactions on Information Systems. [21] Huang, X., Li, S., Yu, M., Sesia, M., Hassani, H., Lee, I., Bastani, O., and Dobriban, E. (2024b). Uncertainty in Language Models: Assessment through Rank-Calibration. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 284–312. [22] Ielanskyi, M., Schweighofer, K., Aichberger, L., and Hochreiter, S. (2025). Addressing Pitfalls in the Evaluation of Uncertainty Estimation Methods for Natural Language Generation. In ICLR Workshop: Quantify Uncertainty and Hallucination in Foundation Models: The Next Frontier in Reliable AI. [23] Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y., Chen, D., Dai, W., Chan, H. S., Madotto, A., and Fung, P. (2023). Survey of Hallucination in Natural Language Generation. ACM Computing Surveys, 55(12):1–38. [24] Joshi, M., Choi, E., Weld, D., and Zettlemoyer, L. (2017). TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601– 1611. [25] Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., HatfieldDodds, Z., DasSarma, N., Tran-Johnson, E., et al. (2022). Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221. [26] Kang, S., Bakman, Y. F., Yaldiz, D. N., Buyukates, B., and Avestimehr, S. (2025). Uncertainty quantification for hallucination detection in large language models: Foundations, methodology, and future directions. arXiv 2510.12040. 11
[27] Kuhn, L., Gal, Y., and Farquhar, S. (2023). Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. International Conference on Learning Representations. [28] Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., Toutanova, K., Jones, L., Kelcey, M., Chang, M.-W., Dai, A. M., Uszkoreit, J., Le, Q., and Petrov, S. (2019). Natural Questions: A Benchmark for Question Answering Research. Transactions of the Association for Computational Linguistics, 7:452–466. [29] Lee, K., Lee, K., Lee, H., and Shin, J. (2018). A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in neural information processing systems, 31. [30] Lin, C.-Y. (2004). ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81. Association for Computational Linguistics. [31] Lin, Z., Trivedi, S., and Sun, J. (2023). Generating with confidence: Uncertainty quantification for black-box large language models. Findings of ACL. [32] Lin, Z., Trivedi, S., and Sun, J. (2024a). Contextualized Sequence Likelihood: Enhanced Confidence Scores for Natural Language Generation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 10351–10368. [33] Lin, Z., Trivedi, S., and Sun, J. (2024b). Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models. Transactions on Machine Learning Research. [34] Malinin, A. and Gales, M. (2021). Uncertainty estimation in autoregressive structured prediction. International Conference on Learning Representations. [35] Moskvoretskii, V., Marina, M., Salnikov, M., Ivanov, N., Pletenev, S., Galimzianova, D., Krayko, N., Konovalov, V., Nikishina, I., and Panchenko, A. (2025). Adaptive Retrieval Without SelfKnowledge? Bringing Uncertainty Back Home. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6355–6384. [36] Nikitin, A. V., Kossen, J., Gal, Y., and Marttinen, P. (2024). Kernel Language Entropy: Finegrained Uncertainty Quantification for LLMs from Semantic Similarities. In Advances in Neural Information Processing Systems. [37] Niu, C., Wu, Y., Zhu, J., Xu, S., Shum, K., Zhong, R., Song, J., and Zhang, T. (2024). RAGTruth: A hallucination corpus for developing trustworthy retrieval-augmented language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10862–10878. [38] Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318. [39] Qiu, X. and Miikkulainen, R. (2024). Semantic Density: Uncertainty Quantification for Large Language Models through Confidence Measurement in Semantic Space. In Advances in Neural Information Processing Systems. [40] Ren, J., Fort, S., Liu, J., Roy, A. G., Padhy, S., and Lakshminarayanan, B. (2021). A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detection. arXiv:2106.09022 [cs]. [41] Sahoo, P., Meharia, P., Ghosh, A., Saha, S., Jain, V., and Chadha, A. (2024). A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 11709–11724. [42] Santilli, A., Golinski, A., Kirchhof, M., Danieli, F., Blaas, A., Xiong, M., Zappella, L., and Williamson, S. (2025). Revisiting Uncertainty Quantification Evaluation in Language Models: Spurious Interactions with Response Length Bias Results. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 743– 759. 12
[43] Sriramanan, G., Bharti, S., Sadasivan, V. S., Saha, S., Kattakinda, P., and Feizi, S. (2024). LLM-Check: Investigating Detection of Hallucinations in Large Language Models. In NeurIPS. [44] Tian, K., Mitchell, E., Zhou, A., Sharma, A., Rafailov, R., Yao, H., Finn, C., and Manning, C. (2023). Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 5433–5442. [45] Tomov, T., Fuchsgruber, D., Wollschläger, T., and Günnemann, S. (2026). The Illusion of Certainty: Uncertainty Quantification for LLMs Fails under Ambiguity. arXiv:2511.04418 [cs]. [46] van der Poel, L., Cotterell, R., and Meister, C. (2022). Mutual Information Alleviates Hallucinations in Abstractive Summarization. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 5956–5965. [47] Vashurin, R., Fadeeva, E., Vazhentsev, A., Rvanova, L., Vasilev, D., Tsvigun, A., Petrakov, S., Xing, R., Sadallah, A., Grishchenkov, K., Panchenko, A., Baldwin, T., Nakov, P., Panov, M., and Shelmanov, A. (2025a). Benchmarking uncertainty quantification methods for large language models with LM-polygraph. Transactions of the Association for Computational Linguistics, 13:220–248. [48] Vashurin, R., Goloburda, M., Ilina, A., Rubashevskii, A., Nakov, P., Shelmanov, A., and Panov, M. (2025b). CoCoA: A Minimum Bayes Risk Framework Bridging Confidence and Consistency for Uncertainty Quantification in LLMs. In NeurIPS. [49] Vazhentsev, A., Kuzmin, G., Tsvigun, A., Panchenko, A., Panov, M., Burtsev, M., and Shelmanov, A. (2023). Hybrid Uncertainty Quantification for Selective Text Classification in Ambiguous Tasks. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11659–11681. [50] Vazhentsev, A., Rvanova, L., Kuzmin, G., Fadeeva, E., Lazichny, I., Panchenko, A., Panov, M., Baldwin, T., Sachan, M., Nakov, P., and Shelmanov, A. (2025). Uncertainty-Aware Attention Heads: Efficient Unsupervised Uncertainty Quantification for LLMs. arXiv 2505.20045. [51] Wang, X., Zhang, Z., Chen, G., Li, Q., Luo, B., Han, Z., Wang, H., Li, Z., Gao, H., and Hu, M. (2025). UBench: Benchmarking Uncertainty in Large Language Models with Multiple Choice Questions. In Findings of the Association for Computational Linguistics, pages 8076–8107. [52] Wei, J., Karina, N., Chung, H. W., Jiao, Y. J., Papay, S., Glaese, A., Schulman, J., and Fedus, W. (2024). Measuring short-form factuality in large language models. arXiv:2411.04368 [cs]. [53] Williams, R. J. and Zipser, D. (1989). A Learning Algorithm for Continually Running Fully Recurrent Neural Networks. Neural Computation, 1(2):270–280. [54] Wu, X., Li, X., Quan, L., and Hu, Q. (2025). UncertaintyZoo: A Unified Toolkit for Quantifying Predictive Uncertainty in Deep Learning Systems. arXiv:2512.06406 [cs]. [55] Yang, Q., Ravikumar, S., Schmitt-Ulms, F., Lolla, S., Demir, E., Elistratov, I., Lavaee, A., Lolla, S., Ahmadi, E., Rus, D., Amini, A., and Perez, A. (2023). Uncertainty-aware Language Modeling for Selective Question Answering. arXiv:2311.15451 [cs]. [56] Yao, Y., Wu, H., Guo, Z., Biyan, Z., Gao, J., Luo, S., Hou, H., Fu, X., and Song, L. (2024). Learning From Correctness Without Prompting Makes LLM Efficient Reasoner. In COLM. [57] Yoo, K., Kim, J., Jang, J., and Kwak, N. (2022). Detection of Adversarial Examples in Text Classification: Benchmark and Baseline via Robust Density Estimation. In Findings of the Association for Computational Linguistics, pages 3656–3672. [58] Zha, Y., Yang, Y., Li, R., and Hu, Z. (2023). Alignscore: Evaluating factual consistency with a unified alignment function. arXiv preprint arXiv:2305.16739. [59] Zhang, C., Liu, F., Basaldella, M., and Collier, N. (2024). LUQ: Long-text Uncertainty Quantification for LLMs. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N., editors, Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 5244–5262. 13
[60] Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., Du, Y., Yang, C., Chen, Y., Chen, Z., Jiang, J., Ren, R., Li, Y., Tang, X., Liu, Z., Liu, P., Nie, J.-Y., and Wen, J.-R. (2025). A Survey of Large Language Models. arXiv:2303.18223 [cs]. [61] Zhou, H., Wan, X., Proleev, L., Mincu, D., Chen, J., Heller, K. A., and Roy, S. (2023). Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineering. In ICLR.
A
Related works
A.1
Evaluating uncertainty estimation methods
Work on evaluating uncertainty estimation (UE) methods for LLMs has addressed three concerns: building infrastructure that places heterogeneous methods on a common framework; assessing whether comparisons drawn under controlled conditions hold up in deployment; and questioning whether the evaluation protocols themselves are sound. We add a fourth: whether the evaluation target – correctness as commonly operationalized – adequately reflects the phenomenon (hallucination) that uncertainty is supposed to detect. Method-coverage frameworks. The most direct response to method heterogeneity has been to consolidate estimators behind shared interfaces and shared intermediate quantities. LM-P OLYGRAPH [47] implements a comprehensive estimator suite behind a single pipeline that computes greedy responses, token probabilities, sampled responses, and semantic relation matrices once and reuses them across estimators, controlling for implementation differences. Adjacent toolkits – UQLM [6], U NCERTAINTY Z OO [54], and UB ENCH [51] – provide complementary coverage or standardize estimator responses to a common confidence range. Across these frameworks, the contribution is comparability rather than validity: each evaluation is only as meaningful as its target for correctness. Deployment- and protocol-level critiques. A second strand asks whether rankings produced under controlled benchmarks transfer to deployment, and whether the protocols themselves can be trusted. Bakman et al. [3] evaluate nineteen UE methods under decision-threshold drift, prompt perturbation, long-form adaptation, and ensembling, and find that most methods are sensitive to threshold choice and vulnerable to adversarial prompts – patterns invisible under standard evaluation conditions. Santilli et al. [42] show that the same nuisance factor can bias both UE scores and correctness functions – most strikingly response length – so that a trivial length baseline can rival principled methods under length-biased correctness metrics. Ielanskyi et al. [22] reinforce this concern by showing that the choice of approximate correctness function (ROUGE, BLEURT, LLM-as-judge variants) substantially reorders method rankings, and propose marginalising over multiple judges as a more robust alternative. Complementary critiques target the limited ecological validity of QA-style evaluations [12], the inability of realisable benchmarks to test calibration over inherently variable outcomes [10, 45], and the incomparable scales on which estimator responses live [21]. The shared message is that “method A outperforms method B on benchmark X” is routinely confounded by length, choice of correctness function, threshold calibration, or estimator response scale – and that fixing the benchmark itself does not fix the comparison. The evaluation gap we address. Across these strands, a common limitation is that the evaluation target is rarely a hallucination-specific signal: correctness is operationalized through lexical or semantic overlap with a reference answer, or through a multiple-choice answer key, rather than through an explicit characterization of when an response is unsupported by the evidence the model was supposed to use. Our contribution sits in this space: we retain the LM-P OLYGRAPH estimator catalog and the methodological caution of Santilli et al. [42], Bakman et al. [3], and Ielanskyi et al. [22], but replace generic correctness targets with hallucination targets defined through operational distinctions established in the hallucination literature, which we develop next. A.2
Hallucination: definitions and benchmarks
The definition of hallucination has shifted as LLMs have moved from text-conditioned to largely open-ended generation, and the choice of definition directly shapes what a benchmark measures. 14
From NLG taxonomy to LLM-specific definitions. The intrinsic/extrinsic distinction was originally formulated for task-conditioned NLG by Ji et al. [23]: an response is intrinsically hallucinated if it contradicts an input source, and extrinsically hallucinated if it cannot be verified from that source – a notion that presupposes a single explicit input. As LLMs increasingly generate free-form text without such a source, two reframings followed. Huang et al. [20] proposed factuality versus faithfulness as the LLM-era replacement axis (correctness against the world vs. adherence to context), while Augenstein et al. [2] argued that factuality and hallucination should be treated as distinct concepts entirely: factuality requires an external oracle, whereas hallucination admits an internal definition – whether the response matches the evidence the model was supposed to use. The separation is consequential because the two concepts overlap but are not identical: a model can be factually correct while unfaithful to its evidence, or factually wrong while faithful to a misleading source. HalluLens and the operational extrinsic/intrinsic split. Bang et al. [4] consolidate this picture and introduce the H ALLU L ENS benchmark. They redefine intrinsic hallucinations as responses unsupported by information explicitly provided at inference time, and extrinsic hallucinations as responses unsupported by the pretraining knowledge the model is expected to have encountered. The reformulation is operational: each regime corresponds to a concrete source of evidence against which an response can be adjudicated, with ground-truth labels defined without implicit appeal to world truth. H ALLU L ENS instantiates this with three extrinsic tasks anchored to Wikipedia, along with items generated on-the-fly to mitigate test-set leakage. The Wikipedia anchor rests on a documented approximation – recent pretraining mixtures are assumed to contain Wikipedia, so inconsistency with a Wikipedia article is treated as a proxy for inconsistency with pretraining knowledge – imperfect for long-form generation [16] but the most operationalized proxy currently in use. Benchmark heterogeneity and our positioning. With the operational split in hand, the existing landscape can be repartitioned by the definition each benchmark encodes. Explicitly intrinsic benchmarks include RAGT RUTH [37], which provides word-level hallucination annotations for retrieval-augmented generation. Closed-book QA benchmarks such as T RIVIAQA [24], NATURAL Q UESTIONS [28], and S IMPLE QA [52] target some dimension of factuality but sit ambiguously with respect to the operational split: classifying them as extrinsic-regime benchmarks would require evidence that all their questions are answerable from the model’s pretraining data, which is generally not established. H ALLU L ENS fills this gap by restricting questions to those generated from Wikipedia, making the pretraining-coverage assumption a documented approximation rather than an implicit one; explicitly extrinsic benchmarks otherwise remain comparatively rare. As a result, comparing performance across these benchmarks partly compares definitions rather than methods. We therefore pair RAGT RUTH (intrinsic regime; word-level labels produced before the operational split was formalized, hence not biased toward any one definitional choice) with the three H ALLU L ENS tasks – P RECISE W IKI QA, L ONG W IKI, and N ON E XISTENT R EFUSAL – as the most explicit operational extrinsic benchmarks currently available with controlled test-set generation. Together, these commit us to the hallucination/factuality distinction articulated by Augenstein et al. [2] and operationalized by Bang et al. [4], and yield measurements that remain comparable across regimes without conflating them.
B
Description and implementation details on uncertainty estimators
This appendix specifies the uncertainty estimators evaluated in our benchmark, briefly described in Section 2. We pursue two goals jointly: a conceptual map organized by the signal each method exploits, and an implementation-faithful description of the exact variants executed in our pipeline. Every formula below was derived by auditing the corresponding implementation in our LM-P OLYGRAPHbased pipeline [15, 47] rather than from the high-level descriptions in the original papers. Pipeline. We use LM-P OLYGRAPH as the uncertainty-score computation engine. The framework separates two stages: shared intermediate quantities (greedy responses, token log-probabilities, stochastic samples, semantic relation matrices, internal representations) are computed once per input, and the estimators consume these quantities to produce the final scores. We do not modify any estimator implementation. The pinned version of LM-P OLYGRAPH we use is main/0e5bcdd35efe7de4e3ae8124427aa12575a1bd2c. Downstream evaluation is performed in our analysis pipeline, externally to LM-P OLYGRAPH. 15
B.1
Implementation details
Table 2 consolidates the numerical hyperparameters and external models used in the executed pipeline. Default LM-P OLYGRAPH settings are used for any value not listed. B.2
Code availability
The code is available here: https://anonymous.4open.science/r/uncertainty-benchmark-867C. B.3
Compute resources
All GPU-demanding computation was performed on a university HPC cluster using usage-constrained GPU allocations. The primary hardware was an NVIDIA RTX PRO 6000 Blackwell (96 GB GDDR7) for large-model runs, with NVIDIA RTX 6000 Ada/RTX A6000 (48 GB GDDR6) nodes used for smaller models. Nodes were equipped with 24–32 CPU cores and 192–512 GB RAM. The two main GPU-demanding phases totaled approximately 700 productive GPU-hours: • Benchmark inference (3 tasks × 3 models, up to 2,000 samples each, 50 uncertainty estimators): roughly 415 GPU-hours for HalluLens tasks and 255 GPU-hours for RAGTruth, ranging from 5 GPU-hours per run (small datasets, 7B models) to 87 GPU-hours (large datasets, 13B models). 48 GB of VRAM suffices for 7B models at a batch size of 1; 13B models require 96 GB. • LongWiki evaluation (Llama-3.1-70B-AWQ judge via vLLM): 31 productive GPU-hours across 2 model runs, using 2 GPUs simultaneously (judge server + evaluation client). B.4
Notation and conventions
Let x denote the input query and ŷ = (ŷ1 , . . . , ŷT ) the response to be scored of length T . In the standard free-generation setting, ŷ is the greedy response of the model; in the RAGTruth dataset, where responses have already been generated and annotated, ŷ is the observed response supplied by the dataset. The autoregressive conditional probability of the token ŷt is pθ (ŷt | x, ŷ<t ), and the unconditional language-model probability (input x removed) is pθ (ŷt | ŷ<t ). The full predictive distribution at position t lives on the simplex ∆(V) over the vocabulary of size V = |V|, i.e. QT pθ (· | x, ŷ<t ) ∈ ∆(V). The full sequence probability is Pθ (ŷ | x) = t=1 pθ (ŷt | x, ŷ<t ). P Shannon entropy of any q ∈ ∆(V) is H(q) = − v∈V q(v) log q(v), and U(V) denotes the uniform distribution on V. When stochastic sampling is required, we draw S = 10 responses y (1) , . . . , y (S) ∼ pθ (· | x) at sampling temperature 1 with random seed 42. We write |y (s) | for the length of sample s and Q|y(s) | (s) (s) Pθ (y (s) | x) = t=1 pθ (yt | x, y<t ) for its sequence probability. The same sample pool is shared across all sample-based estimators for a given input, so any two such estimators are compared on identical samples. For semantic-consistency methods, NLI-based pairwise scores are Eij = pNLI (entailment | y (i) , y (j) ),
Cij = pNLI (contradiction | y (i) , y (j) ),
produced by microsoft/deberta-large-mnli [19].4 Cross-encoder semantic similarity, when used, is computed with cross-encoder/stsb-roberta-large.5 Score orientation. All estimators are reported as uncertainty scores: larger values indicate greater uncertainty. Two estimators in LM-P OLYGRAPH are confidence-oriented (Rényi Divergence and Fisher–Rao Distance) and are negated before analysis. The formulas below are uncertainty-oriented. Name versus executed formula. Two estimator names refer to a probability, whereas the executed scores are log-transformations of that probability. We make this explicit to avoid confusion. Maximum Sequence Probability refers to the quantity Pθ (ŷ | x), but the implemented score in LM-P OLYGRAPH is its negative negative log-likelihood (NLL). Similarly, Perplexity refers to P log – the sequence exp − T1 t log pθ (ŷt | x, ŷ<t ) , whereas the executed score is the logarithm of this perplexity, i.e. the length-normalized sequence negative log-likelihood (LN-NLL). 4 https://huggingface.co/microsoft/deberta-large-mnli 5 https://huggingface.co/cross-encoder/stsb-roberta-large
16
Definitional convention. We use := for the exact quantity returned by the implementation and = for subsequent algebraic rewrites. Configuration convention. Unless stated otherwise, we use the default LM-P OLYGRAPH configuration. When a hyperparameter materially affects the score, we report its value, see Table 2. B.5
White-box estimators
White-box estimators require access to information beyond the generated text of the response: the model’s logits, attention maps, hidden states, or training data. We organize them into five families, ordered from the most direct readouts of the predictive distribution to the most indirect: (i) information-based estimators, which aggregate token-level distributional quantities computed in a single forward pass; (ii) sample-dispersion estimators, which lift these quantities to multiple stochastic samples and combine them with measures of inter-sample dispersion; (iii) internal-state estimators, which use attention or hidden-state geometry; (iv) training-based estimators, which score representation-space distance from a fitted reference distribution; and (v) reflexive estimators, which prompt the model to judge its own answer. B.5.1
Information/Logit-based estimators
These estimators consume only the token-level predictive distributions encountered along a single response ŷ and aggregate a position-wise quantity across the sequence. Within this family we distinguish methods that look only at the probability of the realized token (Maximum Sequence Probability, Perplexity), methods that examine the full predictive distribution at each position (Mean Token Entropy, SelfCertainty, Rényi Divergence, Fisher–Rao Distance), methods that contrast the conditional predictive distribution with its unconditional counterpart (Pointwise Mutual Information and its conditional variant), and two methods that introduce auxiliary signals at the token level (TokenSAR, Claim-Conditioned Probability). Maximum Sequence Probability. Maximum Sequence Probability is the most basic logit-based estimator: it considers only the probability assigned to the realized response and ignores the rest of the predictive distribution. The executed score is the sequence negative log-likelihood, UMSP (x; ŷ) := −
T X
log pθ (ŷt | x, ŷ<t ) = − log Pθ (ŷ | x).
t=1
The score is zero when the model assigns probability 1 to every realized token and grows without bound as confidence in the realized response drops. It is length-sensitive: longer responses accumulate more uncertainty at constant per-token confidence. Perplexity. Perplexity removes the length sensitivity of Maximum Sequence Probability by averaging the per-token negative log-likelihood, T
UPPL (x; ŷ) := −
1X log pθ (ŷt | x, ŷ<t ), T t=1
which equals the logarithm of the standard perplexity of ŷ under the model. Like Maximum Sequence Probability, this score uses only the probability of the realized token, but it is comparable across responses of different lengths [18]. Mean Token Entropy. Mean Token Entropy is the simplest logit-based estimator that uses the full predictive distribution at each position rather than only the probability of the realized token. The score averages the Shannon entropy of the predictive distribution across positions, which equals the chain-rule expansion of the sequence entropy [9, 34]: T
UMTE (x) :=
1X H pθ (· | x, ŷ<t ) . T t=1
Unlike MSP and PPL, MTE is not determined by the realized token alone: even when ŷt is itself among the most probable tokens, MTE can still be large if probability mass is spread broadly across 17
alternatives; conversely, when ŷt has only moderate probability but the distribution is peaked on a few competing tokens, MTE remains small because the distribution is concentrated. SelfCertainty. SelfCertainty compares a uniform reference distribution to the model’s predictive distribution, using the reverse KL direction relative to the entropy-like divergence-from-uniform scores used by Rényi Divergence and Fisher–Rao Distance below. Let Ṽt ⊆ V be the set of vocabulary entries with finite log-probability at position t. The implemented token-level quantity is 1 X log pθ (v | x, ŷ<t ), DKL U(Ṽt ) ∥ pθ (· | x, ŷ<t ) = − log |Ṽt | − |Ṽt | v∈Ṽt
and the sequence-level uncertainty score reverses the polarity by taking the negative mean: T
USC (x) := −
1X DKL U(Ṽt ) ∥ pθ (· | x, ŷ<t ) . T t=1
A flat predictive distribution yields a small divergence and USC ≈ 0 (high uncertainty); a sharply peaked distribution yields a large divergence and USC ≪ 0 (low uncertainty). Unlike entropy or Rényi-style scores, the reverse-KL direction is especially sensitive to tokens assigned very small probability: a single very small pθ (v | ·) blows up one term of the sum. Rényi Divergence. Rényi Divergence also compares the predictive distribution to a uniform reference, but in the opposite direction from SelfCertainty – and on a temperature-scaled version of the distribution. Let qt (v | x, ŷ<t ; τ ) = softmax(log pθ (· | x, ŷ<t ) / τ )(v) ∈ ∆(V). The token-level Rényi divergence of order α from qt to the uniform distribution is X 1 log qt (v | x, ŷ<t ; τ )α − log |V|, Dα qt (· | x, ŷ<t ; τ ) ∥ U(V) = α−1 v∈V
and the sequence-level uncertainty score, after polarity reversal, is URenyi (x; τ ) := −
T 1X Dα qt (· | x, ŷ<t ; τ ) ∥ U(V) . T t=1
Larger divergence corresponds to a sharper, more concentrated distribution, hence the negation. The parameter α controls sensitivity to the distribution: α < 1 gives more weight to low-probability tokens, α > 1 emphasizes the peak; the temperature τ smooths the distribution before the divergence is computed. SelfCertainty should not be read as the α = 1 case of this estimator: it uses the KL direction U ∥ p, whereas this Rényi score uses q ∥ U, which measures a different geometric quantity (peak concentration rather than coverage). We use α = 0.5 and τ = 2 [11]. Fisher–Rao Distance. Fisher–Rao Distance also measures how far the temperature-scaled predictive distribution qt (· | x, ŷ<t ; τ ) lies from the uniform distribution on ∆(V), but uses Fisher–Rao geometry rather than a divergence. The token-level geodesic distance is ! Xq 2 1 qt (v | x, ŷ<t ; τ ) · |V| , dFR qt (· | x, ŷ<t ; τ ), U (V) = arccos π v∈V
and the sequence-level uncertainty score, after polarity reversal, is T 1X UFR (x; τ ) := − dFR qt (· | x, ŷ<t ; τ ), U (V) . T t=1
The geodesic distance is zero when qt is uniform and approaches 1 as qt becomes more peaked; the negation orients the score so that a flatter (more uncertain) distribution receives a larger value. Unlike any divergence, Fisher-Rao is a true metric on the simplex (symmetric, satisfies the triangle inequality). We use τ = 2 [11]. 18
Mean Pointwise Mutual Information. The estimators above measure either how confident the model is in the realized response (MSP, PPL) or how broadly its predictive distribution spreads (MTE, SelfCertainty, Rényi, Fisher–Rao). They cannot detect responses that are likely under the model regardless of the input, that is, responses whose probability does not benefit from prompt grounding. Mean Pointwise Mutual Information addresses this by contrasting, for each realized token, the conditional log-probability log pθ (ŷt | x, ŷ<t ) with the unconditional log-probability log pθ (ŷt | ŷ<t ). The latter is obtained by an additional forward pass on the response alone, with the prompt x removed. The executed score is UPMI (x; ŷ) := −
T 1 X log pθ (ŷt | x, ŷ<t ) − log pθ (ŷt | ŷ<t ) . T t=1
When the conditional probability exceeds the unconditional, the prompt is informative and UPMI is negative (low uncertainty); when the two are comparable, the response could have been generated without the prompt and UPMI approaches zero (high uncertainty), which is interpreted as a signal of weak grounding [46]. Mean Conditional Pointwise Mutual Information. Mean Conditional Pointwise Mutual Information refines PMI by gating the unconditional correction: the correction is applied only at positions where the predictive distribution is locally uncertain, and is ignored at sharp positions, where the unconditional log-probability would add noise rather than signal. “Locally uncertain” is operationalized through the same per-position entropy used by MTE: the correction is applied only when H(pθ (· | x, ŷ<t )) ≥ τ , with weight λ. The executed score is UCPMI (x; ŷ) := −
T i 1 Xh log pθ (ŷt | x, ŷ<t ) − λ · 1 H(pθ (· | x, ŷ<t )) ≥ τ · log pθ (ŷt | ŷ<t ) . T t=1
At low-entropy positions, the score reduces to the realized-token negative log-likelihood (Perplexitylike behavior); at high-entropy positions, it behaves like PMI [46]. We use the LM-P OLYGRAPH defaults τ = 0.0656 and λ = 3.599. TokenSAR. TokenSAR re-weights the per-token negative log-likelihoods by an external measure of token relevance, so that tokens whose removal changes the meaning of the response contribute more to the score than function words or filler. Token relevance is computed by leave-one-out cross-encoder similarity: for each position t, let simt ∈ [0, 1] be the semantic similarity between ŷ and the response with token ŷt removed. The unnormalized relevance is 1 − simt , and the normalized relevance is PT Rt = (1 − simt )/ t′ =1 (1 − simt′ ). The executed score is UTSAR (x; ŷ) :=
T X
Rt · − log pθ (ŷt | x, ŷ<t ) .
t=1
Only high-relevance improbable tokens drive uncertainty; an improbable but semantically dispensable token has little effect [13]. Claim-Conditioned Probability. Claim-Conditioned Probability targets a different failure mode of standard token-probability estimators: at each position, several alternative tokens may be plausible substitutes, and only some of them would change the meaning of the response. The executed pipeline proceeds in three steps. At each position t, the top-10 token alternatives At = {at,0 , at,1 , . . . , at,9 } are identified, where at,0 = ŷt is the greedy token itself. Each alternative at,i is then mapped to a candidate response by substituting it at position t in ŷ, and bidirectional NLI is evaluated between the greedy-substituted response (i.e., ŷ itself) and the at,i -substituted response. The two directional labels are combined into a single label ct,i ∈ {entail, contra, neutral}: if both directions agree, that label is used; if one direction is entail and the other contra, the label is neutral; otherwise the single non-neutral label is used. Alternatives with ct,i = entail and the greedy alternative at,0 (which is trivially entailing) form the entailment set Et , alternatives with ct,i = contra form the contradiction set Ct , and neutral alternatives are discarded. At position t, the claim-conditioned probability is the share of probability mass on entailing alternatives within the entailing-or-contradicting subset, P pθ (a | x, ŷ<t ) CCPt = P a∈Et , a∈Et ∪Ct pθ (a | x, ŷ<t ) 19
and the sequence-level uncertainty score is the negated product across positions, UCCP (x) := −
T Y
CCPt .
t=1
Since neutral alternatives are excluded from the denominator, the ratio measures semantic support conditional on the alternative being either entailing or contradicting, not support against the full predictive distribution. A single position at which probability leaks toward contradicting alternatives is enough to pull the product toward zero and the score toward zero in absolute value (high uncertainty); concentration of mass on entailing alternatives produces a near-1 product and a score near −1 (low uncertainty) [14]. B.5.2
Sample semantic dispersion estimators
Sample-dispersion estimators draw S stochastic samples y (1) , . . . , y (S) from the predictive distribution and combine sample-level probabilities with a measure of dispersion across samples. Beyond the probabilities themselves, several estimators in this family use auxiliary signals (semantic relations from an NLI model, semantic similarity from a cross-encoder, or internal embeddings); we characterize them as probability-aware sample-based estimators rather than as pure consistency estimators. They mirror the structure of the logit-based family: probability-only methods (Monte Carlo Sequence Entropy and its length-normalized variant), methods that summarize agreement under a notion of meaning (Semantic Entropy, Semantic Density), and hybrid methods that combine probability with an external relevance or consistency signal (SentenceSAR, SAR, the Cocoa family). Monte Carlo Sequence Entropy. Monte Carlo Sequence Entropy approximates the model’s expected sequence negative log-likelihood under its own sampling distribution: (s)
S S |y | 1X 1XX (s) (s) UMCSE (x) := − log Pθ (y (s) | x) = − log pθ yt | x, y<t . S s=1 S s=1 t=1
A model that consistently produces high-probability samples receives a low score; a model that scatters probability mass across many possible responses receives a high score [34]. Monte Carlo Normalized Sequence Entropy. Monte Carlo Normalized Sequence Entropy is the length-normalized version of the previous estimator: each sample’s negative log-likelihood is divided by its own length before averaging, so that the score is comparable across prompts that elicit responses of different lengths: S
1X 1 UMCNSE (x) := − log Pθ (y (s) | x). S s=1 |y (s) | Semantic Entropy. Semantic Entropy uses the same sample-based negative log-likelihoods, but groups them by meaning before computing entropy, so that paraphrases of the same answer do not inflate the score. The pipeline first partitions the samples into semantic equivalence classes C1 , . . . , CM using bidirectional NLI entailment: y (i) and y (j) are placed in the same class when they entail one another. The probability of class Cm is then obtained by summing sample probabilities and renormalizing: P (s) | x) s : y (s) ∈Cm Pθ (y P (Cm | x) = . PS (s) | x) s=1 Pθ (y The uncertainty score is the entropy of this class distribution: USE (x) := −
M X
P (Cm | x) log P (Cm | x).
m=1
A model that always produces the same meaning regardless of surface form yields a single class and zero entropy; a model that distributes mass over several meanings yields large entropy [17, 27]. 20
Semantic Density. Semantic Density replaces the hard equivalence classes of Semantic Entropy with a soft NLI-based kernel and a probability-weighted aggregation around the greedy response ŷ. 1 (s) Let P̃θ (y (s) | x) = exp |y(s) log P (y | x) denote the length-normalized sample probability, and θ | let Ks denote the NLI-based kernel score that quantifies the soft semantic agreement between sample s and the surrounding samples (including ŷ). Writing P̃θ (ŷ | x) for the length-normalized probability of the greedy response, the executed score is the negated probability-weighted kernel density PS (s) | x) Ks + P̃θ (ŷ | x) s=1 P̃θ (y USD (x; ŷ) := − P . S (s) | x) + P̃θ (ŷ | x) s=1 P̃θ (y The score is close to −1 (low uncertainty) when the probable samples cluster in a dense semantic neighborhood of ŷ, and close to 0 (high uncertainty) when the kernel scores are uniformly low [39]. SentenceSAR. SentenceSAR extends the relevance-weighting idea of TokenSAR to the multisample setting. The contribution of each sample to the score is its own probability augmented by a relevance term that aggregates the probabilities of other samples weighted by their semantic similarity to the current one. Writing simsj for the cross-encoder similarity between y (s) and y (j) and τ for a temperature parameter, the support term is 1X RS (s; τ ) = Pθ (y (j) | x) simsj , τ j̸=s
and the executed score is USentSAR (x; τ ) :=
S 1X − log Pθ (y (s) | x) + RS (s; τ ) . S s=1
A sample that is both probable and semantically supported by other samples contributes little to the score; a sample that is improbable and semantically isolated contributes a large negative log [13]. We use τ = 1. SAR. SAR is the full pipeline of Duan et al. [13], combining TokenSAR at the token level and SentenceSAR at the sentence level. The raw sample probability used in SentenceSAR is replaced with the relevance-adjusted probability from TokenSAR. Concretely, with P̃θtsar (y (s) ) = (s) exp −UTSAR (x, y ) the relevance-weighted probability of sample s, the support term becomes P R̃S (s; τ ) = (1/τ ) j̸=s P̃θtsar (y (j) ) simsj , and the executed score is USAR (x; τ ) :=
S 1X − log P̃θtsar (y (s) ) + R̃S (s; τ ) . S s=1
SAR composes two relevance signals – token-level for each sample and sentence-level across samples – into a single uncertainty score. We use τ = 1. Cocoa family. The Cocoa family follows a simple template: a base uncertainty score is multiplied by an inter-sample semantic dissimilarity factor, so that the final score is large only when the response is uncertain on its own and semantically dissimilar to the alternative samples [48]. Let ubase (x; ŷ) denote the base uncertainty and let S sent ∈ RK×K be the cross-encoder sentence-similarity matrix over the K sentence units computed by LM-P OLYGRAPH for this input (greedy response and/or sent samples, depending on the module configuration). The dissimilarity factor is the mean of 1 − Sij across the full matrix, K K 1 XX sent Dsent (x) = 2 1 − Sij , K i=1 j=1 and the executed Cocoa score is the product UCocoa (x; ŷ) := ubase (x; ŷ) · Dsent (x). We instantiate three variants by choosing ubase ∈ {UMSP , UPPL , UMTE }, denoted CocoaMSP, CocoaPPL, and CocoaMTE. An improbable response that is semantically consistent with the alternative 21
samples is treated as a paraphrase and receives a low score; an improbable response that is also semantically isolated receives a high score. Implementation note: although the LM-P OLYGRAPH source comment states that the diagonal is excluded, the executed code averages over the full K × K sent matrix. Since self-similarity is 1, the diagonal entries contribute 0 to 1 − Sij and thus dilute Dsent by a factor of (K − 1)/K. B.5.3
Internal-state estimators
Internal-state estimators use latent computations of the model – attention weights or hidden representations – rather than only the response distribution. Their common premise is that predictive unreliability may already be visible in the model’s internal processing dynamics before it is fully reflected in the response token distribution. AttentionScore. AttentionScore measures self-attention concentration along the generated sequence, assuming that tokens with weak self-attention are less stably represented and therefore more error-prone. Writing αℓ,h (t, t) ∈ [0, 1] for the diagonal self-attention weight at layer ℓ, head h, and position t, the executed score is UAttn (x; ŷ) := −
H T 1 XX log αℓ,h (t, t) + ε , H t=1 h=1
−12
where ε = 10 is a numerical stabilizer and the layer ℓ is the model’s middle layer (ℓ = L/2), as selected by LM-P OLYGRAPH when no layer is explicitly specified. This choice matters: attentionbased diagnostics vary substantially across layers, and the score should be understood as a layerspecific instantiation of the general idea [43]. RAUQ (Recurrent Attention Uncertainty Quantification). RAUQ propagates a confidence signal recursively along the sequence, combining the local token probability with attention from the current token to the previous one, and aggregates the per-layer scores only over the middle third of layers. Let L = {⌊L/3⌋, . . . , ⌈2L/3⌉} denote the indices of the middle third of the model’s L attention layers. For each ℓ ∈ L, the head attending most strongly (on average) to the previous token is h∗ (ℓ) = arg max h
T 1 X αℓ,h (t, t − 1). T − 1 t=2 (ℓ)
The recurrent confidence at layer ℓ is initialized at C1 = pθ (ŷ1 | x) and updated for t > 1 by (ℓ)
(ℓ)
= α · pθ (ŷt | x, ŷ<t ) + (1 − α) · αℓ,h∗ (ℓ) (t, t − 1) · Ct−1 , P (ℓ) where α ∈ [0, 1] is a mixing weight. The per-layer score is 1 − T1 t log Ct , and the estimator returns the worst (largest) layer over the middle third: " # T 1X (ℓ) URAUQ (x; ŷ) := max 1 − log Ct . ℓ∈L T t=1 Ct
Unlike AttentionScore, which evaluates each position independently, RAUQ captures how confidence accumulates or degrades along the sequence [50]. When α is not specified, LM-P OLYGRAPH selects it through two binary flags: 1ent = 1 if the entropy-based variant is used (and 0 for the probability variant), and 1instr = 1 if the underlying model is instruction-tuned (and 0 otherwise). The default is α = (1 − 1ent ) (0.2 + 0.3 · 1instr ) + 1ent (0.8 + 0.1 · 1instr ), which yields α ∈ {0.2, 0.5, 0.8, 0.9} depending on the flags. We use the probability variant (1ent = 0) on instruction-tuned models (1instr = 1), so the executed α = 0.5. Contextualized Sequence Likelihood (CSL). Contextualized Sequence Likelihood reweights token negative log-likelihoods by an attention-derived saliency, so that positions the model deems important when producing the final response contribute more to the score than filler positions. Let ᾱt = meanℓ,h αℓ,h (last, t) be the mean attention paid to position t from the final generation step, 22
averaged P over layers and heads, and let wt = ᾱt / (wt ≥ 0, t wt = 1). The executed score is UCSL (x; ŷ) :=
T X
P
t′ ᾱt′ be the corresponding saliency weight
wt · − log pθ (ŷt | x, ŷ<t ) .
t=1
The score has the same form as TokenSAR but uses an internal saliency signal in place of an external cross-encoder relevance signal [32]. EigenScore. EigenScore measures the geometric spread of sampled internal embeddings in representation space. Let e(s) ∈ Rd denote the decoder hidden state of the last generated token for sample s, and let E = [e(1) , . . . , e(S) ] ∈ Rd×S be the matrix whose columns are the S sample embeddings. d×d Writing Jd = Id − d1 1d 1⊤ for the centering matrix and α for a regularization scalar, the d ∈ R regularized centered covariance matrix is C = E⊤ Jd E + αIS ∈ RS×S , with eigenvalues λ1 , . . . , λS . The executed score is the mean log-eigenvalue, S
UEigen (x) :=
1X log λk . S k=1
Sampled responses whose embeddings cluster in a low-dimensional subspace yield small eigenvalues and a low score (low uncertainty); sampled responses whose embeddings span many directions yield large eigenvalues and a high score (high uncertainty) [7]. We use α = 10−3 . B.5.4
Training-based estimators
Training-based estimators fit a reference distribution in representation space on a held-out training partition and score test inputs by their distance from that reference. They are best read as familiarity or out-of-distribution signals rather than as readouts of response confidence. All variants below operate on decoder hidden states averaged over the generated tokens of ŷ, and are all fitted to each task’s own training split. The relative variants additionally subtract a background distance term fitted on a broad, general-purpose corpus that we describe below. Mahalanobis Distance. Mahalanobis Distance is the basic representation-space estimator. Let e(x, ŷ) ∈ Rd denote the decoder hidden state averaged over all generated tokens of ŷ, and let µ ∈ Rd and Σ ∈ Rd×d be the mean and covariance estimated from the task’s training partition. The executed score is the Mahalanobis distance between e(x, ŷ) and the training centroid: q ⊤ UMD (x; ŷ) := e(x, ŷ) − µ Σ−1 e(x, ŷ) − µ . A test input whose representation lies near the training centroid receives a small distance (low uncertainty); an input far from the training distribution receives a large distance (high uncertainty) [29]. Relative Mahalanobis Distance. Relative Mahalanobis Distance subtracts a background Mahalanobis distance, fitted on a broad general-purpose corpus, in order to isolate task-specific novelty from generic rarity: URMD (x; ŷ) := UMD (x; ŷ) − UMD0 (x; ŷ), where UMD0 uses the background mean and covariance in place of the task mean and covariance. A test input far from both the task and the background receives a small score (general rarity); an input far from the task but close to the background receives a large score, indicating domain shift specific to the task [40]. We fit the background distribution on allenai/c4 (shard en/c4-train.00000-of-01024.json.gz).6 6 https://huggingface.co/datasets/allenai/c4
23
Robust Density Estimation (RDE). Robust Density Estimation replaces the Gaussian fit of Mahalanobis Distance with a robust covariance estimate computed over a Kernel-PCA-projected representation space, to better handle non-elliptical embedding distributions and reduce sensitivity to ′ ′ training-set outliers. Let ẽ(x; ŷ) ∈ Rd denote the Kernel-PCA projection of e(x, ŷ), and let µ̂ ∈ Rd ′ ′ and Σ̂ ∈ Rd ×d be the mean and covariance estimated by the Minimum Covariance Determinant procedure in the projected space. The executed score is the Mahalanobis distance in this robust, dimensionality-reduced space: q ⊤ −1 ẽ(x, ŷ) − µ̂ Σ̂ ẽ(x, ŷ) − µ̂ . URDE (x; ŷ) := [57]. HUQ-MD and HUQ-RMD. The HUQ variants combine a representation-space distance (epistemicnovelty signal) with a probability-based uncertainty (aleatoric signal) via empirical-CDF rank fusion [49]. Because Perplexity and Mahalanobis Distance have different scales and distributions, one cannot combine them by raw averaging; rank fusion instead converts each score to a scale-free percentile relative to the training partition before averaging. Concretely, let Dtrain denote the task’s training partition. The empirical CDF of UPPL on that partition is X 1 train FPPL (u) = 1 UPPL (x′ , ŷ(x′ )) ≤ u ∈ [0, 1], |Dtrain | ′ x ∈Dtrain
train and FMD is defined analogously for UMD . Evaluating these CDFs at a test input gives its percentile rank on each score (the fraction of training inputs with a lower score), and the executed HUQ-MD score is the mean of the two percentile ranks: h i train train UHUQ-MD (x; ŷ) := 12 FPPL UPPL (x; ŷ) + FMD UMD (x; ŷ) ∈ [0, 1].
HUQ-RMD is defined analogously with URMD in place of UMD . Because both components are converted to [0, 1]-valued percentile ranks before averaging, the combination is scale-free and distribution-free. B.5.5
Reflexive estimators (white-box)
Reflexive estimators prompt the model to judge its own answer. The white-box variants additionally require direct access to the probability assigned to the “True” option; their black-box counterpart is described in Section B.6.4. PTrue. Given an input x and a proposed answer ŷ, PTrue constructs a self-evaluation prompt promptPT (x; ŷ) asking the model to judge whether ŷ is correct and reads off the probability assigned to the “True” option. The executed score is UPTrue (x; ŷ) := − log pθ “True” | promptPT (x; ŷ) . A high probability of “True” yields a small score (low uncertainty); a low probability yields a large score (high uncertainty) [25]. The exact prompt template is: Question: {x}. Proposed Answer: {ŷ}. Is the proposed answer True or False?
PTrue Sampling. PTrue Sampling augments the self-evaluation prompt with S sampled candidate answers y (1) , . . . , y (S) before asking the truth judgment, in order to expose the model to its own response variability: UPTrue-S (x; ŷ) := − log pθ “True” | promptPT-S (x, ŷ, y (1) , . . . , y (S) ) . The exact prompt template is: Question: {x} Here are some ideas that were brainstormed: Possible answer: {ŷ} Is the possible answer: (A) True (B) False The possible answer is:
24
{y (1) , . . . , y (S) }
B.6
Black-box estimators
Black-box estimators operate only on sampled responses and external semantic/lexical-relation functions, without using token probabilities, attention, or hidden states. We organize them into four groups: the simplest estimators that summarize the semantic-class structure of the samples (NumSet, LabelProb); spectral and graph-density estimators on a pairwise similarity graph (Kernel Language Entropy, EigValLaplacian, Eccentricity, DegMat, LUQ); lexical baselines that replace semantic relations with pairwise surface overlap (ROUGE-L Similarity, BLEU Similarity); and a reflexive black-box estimator (PTrue Empirical). B.6.1
Semantic-class summaries
NumSet. NumSet is the simplest semantic-consistency estimator: it counts the number of distinct semantic classes among the sampled responses. Samples are partitioned into bidirectional-entailment classes C1 , . . . , CM , as in Semantic Entropy, and the executed score is the cluster count: UNumSet (x) := M. The score is integer-valued, with a minimum of 1 when all samples express the same meaning and a maximum of S when every sample expresses a different meaning [33]. The granularity is coarse, but the score is robust to NLI-model calibration. LabelProb. We retain the package name L ABEL P ROB for consistency with LM-P OLYGRAPH, but the executed implementation is more accurately described as the complement of the dominant semantic-class frequency: |Cm | . ULabelProb (x) := 1 − max 1≤m≤M S The score is zero when a single semantic class accounts for all samples, and approaches 1 − 1/S when every sample belongs to a different class. LabelProb is therefore best understood as a majority-class complement, not as a black-box approximation of maximum sequence probability as the name might suggest [47]. B.6.2
Spectral and graph-density estimators
These estimators build a pairwise relation matrix W = (Wij )Si,j=1 ∈ RS×S over sampled responses and summarize its topology. Three of them rely on spectral analysis of the corresponding graph (Kernel Language Entropy uses kernel eigenvalues; EigValLaplacian uses Laplacian eigenvalues; Eccentricity uses Laplacian eigenvectors), and the remaining two summarize the graph through its degree structure (DegMat) or through average pairwise scores (LUQ). For EigValLaplacian, DegMat, and Eccentricity, we run three edge constructions: NLI-entailment (Wij = Eij ), NLI-contradiction (Wij = 1 − Cij ), and Jaccard token similarity. Kernel Language Entropy (KLE). KLE quantifies semantic uncertainty Pas thevon Neumann entropy of a heat-diffusion kernel on the semantic graph. Let D = diag j Wij be the degree matrix, L = D − W the unnormalized graph Laplacian, and K = exp(−tL) the heat kernel (subsequently row-normalized to unit trace). Writing λk (K) for the eigenvalues of K: X UKLE (x) := −tr K log K = − λk (K) log λk (K). k
A graph with many disconnected semantic clusters yields several large eigenvalues and a high entropy; a fully connected graph yields a single dominant eigenvalue and low entropy [36]. EigValLaplacian. EigValLaplacian also operates on the spectrum of the graph, but on the eigenvalues of the normalized Laplacian L̃ = I − D−1/2 WD−1/2 rather than on a heat kernel. Writing λ1 , . . . , λS ∈ [0, 2] for its eigenvalues: UEVL (x) :=
S X
max(0, 1 − λk ).
k=1
Each term measures how close one mode of variation is to being fully disconnected; the sum is a continuous relaxation of the number of semantic sets [33]. 25
Eccentricity. Eccentricity (Ecc) uses the eigenvectors of L̃ rather than its eigenvalues. Let u1 , . . . , uk be the eigenvectors of L̃ corresponding to its smallest k eigenvalues, and stack their per-sample entries into spectral embeddings vi = u1 [i], . . . , uk [i] ∈ Rk for sample i. Writing P v̄ = S1 i vi for the centroid: UEcc (x) :=
!1/2
S X
∥vi − v̄∥22
.
i=1
A graph in which all samples occupy a tight region of the spectral embedding space yields a small score; a graph with spectral outliers yields a large score [33]. DegMat. Degree Matrix (DegMat) summarizes a graph’s degree structure rather than its spectrum. The executed score is the mean missing pairwise similarity: PS PS S 2 − i=1 j=1 Wij UDegMat (x) := , S2 which equals zero when all pairs have Wij = 1 (full mutual agreement) and approaches 1 when no pairs agree. Like Eccentricity, DegMat depends on the choice of edge construction, and we report all three variants [33]. LUQ. Long-text Ucertainty Quantification (LUQ) summarizes the graph through average pairwise NLI confidence, using the continuous softmax score from the NLI logits rather than discretized E E C entail/contradict decisions. Writing σij = exp(zij )/ exp(zij ) + exp(zij ) for the soft entailment (i) (j) confidence between y and y : ULUQ (x) := 1 −
S X 1 σij . S(S − 1) i,j=1 i̸=j
A high average pairwise entailment confidence yields a small score (low uncertainty); weak average confidence yields a large score [59]. B.6.3
Lexical similarity baselines
ROUGE-L Similarity and BLEU Similarity. These baselines replace NLI-based semantic relations with pairwise surface-overlap measures. For metric(·, ·) ∈ {ROUGE-L, BLEU}, the executed score is the negated mean pairwise similarity: ULexSim (x) := −
2 S(S − 1)
X
metric(y (i) , y (j) ).
1≤i<j≤S
Surface-form variation across samples yields a high score even when the samples convey the same meaning, so these baselines cannot distinguish valid paraphrasing from semantic disagreement [30, 38]. We report ROUGE-L and BLEU variants separately. B.6.4
Reflexive estimator (black-box)
PTrue Empirical. PTrue Empirical is the black-box counterpart of PTrue: instead of reading off the probability of the “True” token, it draws N stochastic self-evaluation responses from the model and measures the empirical frequency with which the model judges its own answer to be true: UPTrue-E (x; ŷ) := 1 −
N i 1 X h 1 “True” ∈ samplen promptPT (x; ŷ) , N n=1
using the same self-evaluation prompt as PTrue. We use N = 10 [44]. 26
Estimator
Samples
Hyperparameters and external models
logit-based (Sec. B.5.1) Rényi Divergence Fisher–Rao Distance CPMI TokenSAR CCP
1 1 1 1 1
α = 0.5, τ = 2 τ =2 τ = 0.0656, λ = 3.599 cross-encoder stsb-roberta-large NLI deberta-large-mnli; top-k alternatives, k = 10
Sample semantic dispersion (Sec. B.5.2) MC-SE, MC-NSE S = 10 Semantic Entropy S = 10 Semantic Density S = 10 SentenceSAR S = 10 SAR S = 10 Cocoa{MSP, PPL, MTE} S = 10
sampling temperature 1 NLI deberta-large-mnli NLI deberta-large-mnli cross-encoder stsb-roberta-large; τ = 1 cross-encoder stsb-roberta-large; τ = 1 cross-encoder stsb-roberta-large;
Internal state (Sec. B.5.3) AttentionScore RAUQ EigenScore
1 1 S = 10
layer ℓ = L/2; ε = 10−12 α = 0.5; middle-third layers L = {⌊L/3⌋, . . . , ⌈2L/3⌉} decoder hidden state of last token; α = 10−3
Training-based (Sec. B.5.4) Mahalanobis Distance Relative MD RDE HUQ-MD, HUQ-RMD
1 1 1 1
decoder hidden states; fitted per task on training partition background: allenai/c4, shard en/c4-train.00000 Kernel-PCA projection + MCD covariance CDF rank fusion of UPPL and UMD / URMD on training partition
Reflexive, white-box (Sec. B.5.5) PTrue 1 PTrue Sampling S = 10
prompt: see §B.5.5 prompt: see §B.5.5
Black-box (Sec. B.6) NumSet, LabelProb KLE EigValLaplacian, Ecc, DegMat LUQ Lexical Similarity PTrue Empirical
NLI deberta-large-mnli NLI deberta-large-mnli; heat-kernel time t = 0.3 NLI deberta-large-mnli; NLI deberta-large-mnli metrics: ROUGE-L, BLEU prompt: see §B.6.4
S = 10 S = 10 S = 10 S = 10 S = 10 N = 10 self-eval.
Table 2: Numerical hyperparameters and external models for every estimator executed in our benchmark. “Samples” is the number of stochastic generations generated per input. Estimators that have no tunable hyperparameters and no external models (MSP, Perplexity, Mean Token Entropy, SelfCertainty, PMI, MC-SE, MC-NSE, CSL) are omitted from the table. Across all sample-based estimators, the sample pool is shared, the sampling temperature is 1, and the random seed is 42.
C
Methodology details
C.1
Asbtention Template Abstention template. “No information is available about this, and I am not aware of any such entity being recognized or documented. This name does not appear to correspond to anything known, and I do not believe this exists as a real or established thing. There may be some confusion, or this could be a misspelling or an alternative rendering of a different name. I am not certain what this refers to, and this term is not widely recognized in any domain I can identify. If you could provide more context or clarify your intent, that would help determine whether a related concept exists.”
D
Complementary experimental results
27
AUROC ↑
RAGTruth PRR ↑
RCE ↓
AUROC ↑
PreciseWiki PRR ↑
RCE ↓
AUROC ↑
LongWiki PRR ↑
RCE ↓
AUROC ↑
NonExistent PRR ↑
RCE ↓
information-based
Maximum Seq. Probability (MSP) Perplexity (PPL) Mean Token Entropy (MTE) Pointwise Mutual Info. (PMI) Conditional PMI Self Certainty Rényi Divergence Fisher–Rao Distance Token SAR Claim-Conditioned Probability (CCP)
0.64±0.10 0.44±0.09 0.52±0.01 0.33±0.01 0.51±0.07 0.46±0.02 0.45±0.02 0.47±0.03 0.44±0.09 0.61±0.11
0.28±0.26 −0.23±0.17 −0.08±0.05 −0.50±0.05 −0.09±0.20 −0.23±0.05 −0.26±0.04 −0.22±0.06 −0.23±0.17 0.23±0.25
0.33±0.24 0.83±0.24 0.65±0.07 0.99±0.01 0.66±0.20 0.82±0.10 0.89±0.08 0.81±0.11 0.83±0.24 0.40±0.30
0.71±0.05 0.65±0.07 0.64±0.07 0.47±0.00 0.31±0.03 0.57±0.02 0.55±0.02 0.55±0.03 0.71±0.05 0.72±0.02
0.56±0.08 0.45±0.26 0.46±0.22 −0.17±0.05 −0.73±0.14 0.34±0.14 0.28±0.12 0.26±0.15 0.57±0.19 0.61±0.01
0.28±0.07 0.33±0.20 0.35±0.17 0.89±0.02 0.99±0.02 0.48±0.05 0.54±0.05 0.55±0.09 0.21±0.10 0.21±0.04
0.56±0.03 0.57±0.03 0.57±0.03 0.51±0.02 0.43±0.03 0.55±0.02 0.54±0.02 0.55±0.02 0.57±0.03 0.62±0.02
0.21±0.06 0.23±0.07 0.21±0.09 0.00±0.09 −0.22±0.11 0.13±0.06 0.10±0.06 0.15±0.07 0.23±0.07 0.40±0.05
0.44±0.13 0.37±0.15 0.40±0.15 0.68±0.12 0.94±0.07 0.47±0.14 0.49±0.13 0.46±0.15 0.36±0.15 0.28±0.08
0.67±0.08 0.43±0.13 0.44±0.13 0.44±0.02 0.57±0.13 0.50±0.08 0.50±0.07 0.53±0.07 0.42±0.15 0.73±0.09
0.38±0.19 −0.32±0.49 −0.26±0.48 −0.11±0.09 0.10±0.36 −0.07±0.28 −0.09±0.27 −0.02±0.24 −0.39±0.58 0.55±0.18
0.38±0.18 0.74±0.21 0.74±0.20 0.78±0.06 0.61±0.31 0.62±0.12 0.59±0.11 0.54±0.06 0.74±0.20 0.30±0.11
Sampling-based
Monte-Carlo Seq. Entropy Monte-Carlo Norm. Seq. Entropy Semantic Entropy Semantic Density Sentence SAR SAR Cocoa MSP Cocoa PPL Cocoa MTE
0.42±0.06 0.37±0.02 0.44±0.06 0.45±0.05 0.51±0.08 0.41±0.05 0.62±0.10 0.45±0.07 0.51±0.02
−0.36±0.14 −0.43±0.10 −0.31±0.15 −0.18±0.17 −0.14±0.21 −0.32±0.12 0.24±0.25 −0.21±0.16 −0.10±0.06
0.87±0.09 1.00±0.00 0.87±0.09 0.87±0.08 0.68±0.19 0.93±0.08 0.39±0.28 0.84±0.22 0.69±0.10
0.67±0.07 0.67±0.06 0.69±0.07 0.75±0.03 0.72±0.06 0.72±0.05 0.74±0.04 0.69±0.06 0.69±0.06
0.48±0.18 0.49±0.18 0.50±0.16 0.61±0.10 0.59±0.10 0.58±0.12 0.63±0.04 0.55±0.17 0.57±0.15
0.27±0.11 0.27±0.14 0.29±0.07 0.24±0.15 0.27±0.06 0.24±0.05 0.23±0.02 0.26±0.12 0.23±0.08
0.52±0.01 0.53±0.01 0.54±0.03 0.59±0.02 0.51±0.02 0.59±0.02 0.61±0.03 0.62±0.04 0.62±0.04
0.05±0.05 0.06±0.05 0.12±0.09 0.29±0.09 0.05±0.06 0.23±0.04 0.33±0.09 0.35±0.09 0.34±0.09
0.61±0.06 0.58±0.09 0.54±0.10 0.32±0.07 0.67±0.11 0.38±0.06 0.30±0.14 0.25±0.10 0.24±0.11
0.58±0.06 0.48±0.08 0.58±0.06 0.42±0.07 0.66±0.05 0.43±0.08 0.60±0.08 0.39±0.12 0.40±0.12
0.23±0.15 −0.09±0.27 0.21±0.16 −0.32±0.22 0.47±0.09 −0.23±0.27 0.22±0.25 −0.45±0.47 −0.39±0.46
0.52±0.14 0.78±0.24 0.55±0.15 0.89±0.09 0.31±0.02 0.85±0.17 0.47±0.23 0.87±0.11 0.85±0.13
Internal
Attention Score RAUQ CSL EigenScore
0.69±0.03 0.44±0.09 0.39±0.02 0.44±0.08
0.41±0.09 −0.22±0.17 −0.37±0.03 −0.29±0.22
0.28±0.03 0.82±0.24 0.97±0.04 0.87±0.09
0.59±0.03 0.68±0.09 0.63±0.04 0.62±0.06
0.31±0.03 0.48±0.24 0.40±0.09 0.23±0.15
0.41±0.10 0.34±0.23 0.35±0.10 0.50±0.09
0.47±0.01 0.56±0.03 0.53±0.03 0.47±0.03
−0.12±0.04 0.23±0.06 0.10±0.09 −0.10±0.13
0.87±0.08 0.41±0.15 0.54±0.17 0.84±0.11
0.72±0.11 0.44±0.12 0.47±0.01 0.56±0.10
0.48±0.20 −0.28±0.47 −0.15±0.14 0.11±0.24
0.42±0.06 0.74±0.20 0.83±0.03 0.66±0.24
Training-based
Mahalanobis Distance (MD) Relative MD (RMD) Robust Density Estimation (RDE) HUQ-MD HUQ-RMD
0.27±0.04 0.29±0.03 0.32±0.05 0.44±0.09 0.44±0.08
−0.48±0.15 −0.43±0.08 −0.23±0.14 −0.23±0.17 −0.22±0.17
1.00±0.00 1.00±0.00 0.98±0.03 0.83±0.24 0.83±0.24
0.50±0.02 0.49±0.02 0.51±0.09 0.64±0.07 0.64±0.07
0.03±0.03 0.00±0.04 0.02±0.38 0.45±0.27 0.45±0.27
0.68±0.07 0.74±0.07 0.62±0.28 0.32±0.21 0.32±0.21
0.40±0.04 0.42±0.05 0.49±0.06 0.57±0.03 0.57±0.03
−0.32±0.16 −0.25±0.19 −0.02±0.20 0.23±0.07 0.23±0.07
0.97±0.04 0.93±0.08 0.76±0.28 0.37±0.15 0.37±0.15
0.35±0.18 0.30±0.15 0.36±0.12 0.43±0.13 0.43±0.13
−0.77±0.71 −0.91±0.69 −0.50±0.51 −0.32±0.49 −0.32±0.49
0.81±0.26 0.88±0.17 0.90±0.15 0.74±0.21 0.74±0.21
Reflexive
P(True) P(True) Sampling P(True) Empirical
0.51±0.09 0.50±0.10 0.57±0.08
−0.04±0.21 −0.03±0.25 0.16±0.17
0.63±0.33 0.64±0.33 0.45±0.24
0.51±0.05 0.51±0.03 0.49±0.03
0.10±0.15 0.04±0.15 −0.02±0.06
0.70±0.19 0.63±0.12 0.72±0.11
0.47±0.01 – 0.49±0.00
−0.11±0.04 – −0.04±0.03
0.87±0.06 – 0.80±0.04
0.42±0.15 – 0.45±0.09
−0.43±0.67 – −0.17±0.30
0.76±0.18 – 0.79±0.16
Black-box
NumSet LabelProb Kernel Language Entropy (KLE) EigValLap NLI (Entail) EigValLap NLI (Contra) EigValLap Jaccard Eccentricity NLI (Entail) Eccentricity NLI (Contra) Eccentricity Jaccard DegMat NLI (Entail) DegMat NLI (Contra) DegMat Jaccard LUQ Lexical Similarity (ROUGE-L) Lexical Similarity (BLEU)
0.47±0.01 0.46±0.08 0.43±0.07 0.44±0.08 0.42±0.02 0.36±0.05 0.45±0.09 0.41±0.01 0.54±0.04 0.44±0.08 0.40±0.02 0.38±0.06 0.38±0.03 0.37±0.06 0.40±0.06
−0.09±0.01 −0.15±0.23 −0.23±0.19 −0.20±0.22 −0.17±0.06 −0.43±0.16 −0.16±0.25 −0.17±0.06 0.05±0.06 −0.21±0.22 −0.23±0.07 −0.40±0.19 −0.29±0.11 −0.42±0.16 −0.37±0.17
0.85±0.05 0.74±0.19 0.85±0.12 0.83±0.14 0.98±0.02 0.98±0.03 0.76±0.18 0.98±0.02 0.63±0.16 0.80±0.15 0.99±0.01 0.94±0.06 0.98±0.03 0.97±0.02 0.94±0.06
0.74±0.03 0.71±0.04 0.74±0.04 0.75±0.04 0.67±0.07 0.65±0.02 0.72±0.04 0.73±0.04 0.66±0.04 0.75±0.03 0.68±0.06 0.66±0.03 0.74±0.04 0.67±0.02 0.66±0.03
0.64±0.08 0.52±0.10 0.61±0.12 0.64±0.10 0.44±0.20 0.43±0.08 0.52±0.10 0.62±0.09 0.41±0.11 0.65±0.08 0.47±0.18 0.44±0.07 0.62±0.11 0.46±0.08 0.43±0.08
0.16±0.05 0.25±0.05 0.24±0.13 0.21±0.09 0.32±0.17 0.29±0.08 0.30±0.04 0.20±0.07 0.30±0.08 0.21±0.11 0.30±0.15 0.27±0.08 0.22±0.11 0.27±0.10 0.27±0.06
0.53±0.01 0.54±0.03 0.60±0.03 0.59±0.03 0.65±0.03 0.58±0.02 0.58±0.03 0.64±0.02 0.49±0.01 0.57±0.03 0.65±0.03 0.59±0.02 0.65±0.03 0.55±0.01 0.57±0.02
0.13±0.06 0.12±0.10 0.30±0.09 0.24±0.08 0.41±0.06 0.25±0.05 0.20±0.10 0.37±0.05 −0.02±0.03 0.21±0.09 0.40±0.08 0.26±0.06 0.39±0.08 0.15±0.01 0.21±0.06
0.58±0.03 0.51±0.14 0.30±0.07 0.34±0.08 0.22±0.09 0.36±0.09 0.40±0.09 0.23±0.05 0.78±0.06 0.38±0.10 0.22±0.07 0.34±0.11 0.24±0.03 0.46±0.10 0.37±0.13
0.45±0.01 0.51±0.06 0.47±0.05 0.47±0.05 0.48±0.04 0.39±0.05 0.50±0.05 0.48±0.05 0.52±0.01 0.47±0.06 0.48±0.04 0.39±0.05 0.45±0.04 0.33±0.06 0.30±0.04
−0.24±0.03 0.01±0.15 −0.14±0.10 −0.09±0.17 −0.05±0.11 −0.43±0.19 0.01±0.11 −0.06±0.16 0.08±0.03 −0.07±0.20 −0.09±0.10 −0.43±0.19 −0.21±0.08 −0.66±0.32 −0.79±0.27
0.91±0.05 0.72±0.22 0.82±0.12 0.81±0.14 0.83±0.13 0.97±0.04 0.74±0.16 0.83±0.16 0.62±0.07 0.79±0.17 0.83±0.13 0.96±0.05 0.85±0.10 1.00±0.01 1.00±0.00
AlignScore
Table 3: AUROC ↑, PRR ↑, RCE ↓ for All Models across tasks (fixed metric-wise gradient; warmer = higher value). Uncertainty estimators and AlignScore variants are ranked separately: bold = best per column (ties marked); underline = second-best (ties marked) within the uncertainty group; AlignScore group shows only the best.
AS-Precision(gold_answer) AS-Precision(wikipage) AS-Recall(gold_answer) AS-Recall(wikipage)
0.70±0.03 – 0.55±0.03 –
0.37±0.11 – 0.01±0.09 –
0.51±0.37 – 0.58±0.12 –
0.82±0.06 0.81±0.03 0.90±0.00 0.52±0.03
0.61±0.20 0.83±0.02 0.81±0.01 0.06±0.15
0.54±0.11 0.14±0.03 0.50±0.12 0.68±0.11
0.65±0.03 0.67±0.05 0.59±0.01 0.56±0.01
0.43±0.09 0.50±0.08 0.29±0.07 0.20±0.02
0.23±0.08 0.21±0.11 0.33±0.03 0.39±0.08
0.66±0.08 – 0.65±0.05 –
0.38±0.19 – 0.22±0.22 –
0.42±0.15 – 0.62±0.18 –
Mean AUROC (↑ better)
1.0
0.8 AS-prec-gold AttentionScore MSP CocoaMSP CCP
0.7 0.6
sentSAR
0.5
PPL HUQ-MD tokSAR
0.4
EigVal-J PMI
RMD MD
0.3 0.2
1.0
Information-based Sampling-based Internal State Training-based Reflexive Black-box AlignScore
0.9
Mean AUROC (↑ better)
Uncertainty Metric
0.0
2.5
5.0
7.5
10.0
RDE
12.5
15.0
17.5
20.0
AS-prec-wiki DegMat-e
0.8 0.7
0.4 0.3 CPMI 0.0
2.5
0.8 AS-prec-wiki EigVal-c AS-prec-gold LUQ DegMat-c DegMat-e LabelProb
RDE
Information-based Sampling-based Internal State Training-based Reflexive Black-box AlignScore
0.3 0.2
0.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
22.5
1.0
Mean AUROC (↑ better)
Mean AUROC (↑ better)
0.9
sentSAR 0.5 BB PTrue CPMI RMD 0.4 MD
DegMat-c EigVal-c
Rank std across panels (↓ more consistent)
1.0
0.6
SD sentSAR
RMD PMI
0.5
Rank std across panels (↓ more consistent)
0.7
AS-prec-gold
0.6
0.2
22.5
Information-based Sampling-based Internal State Training-based Reflexive Black-box AlignScore
0.9 AS-recall-gold
20.0
0.8
CCP AttentionScore MSP AS-prec-gold sentSAR
0.7 0.6 0.5
LUQ
PTrue RDE MD RMD
0.4 LexSimROUGE-L LexSimBLEU
0.3 0.2
22.5
Rank std across panels (↓ more consistent)
Information-based Sampling-based Internal State Training-based Reflexive Black-box AlignScore
0.9
0.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
22.5
Rank std across panels (↓ more consistent)
Figure 4: Performance–stability profiles for uncertainty estimators within each task, aggregated across the three models. Each point represents one estimator or AlignScore variant. The vertical axis reports mean ROC-AUC across models for the corresponding task, while the horizontal axis reports the standard deviation of the estimator’s within-model rank; lower values indicate more stable rankings across models. The panels show, from left to right and top to bottom: RAGTruth , PreciseWikiQA , LongWiki , and NonExistentRefusal . AlignScore variants are reference-based comparators rather than uncertainty estimators.
28
AUROC ↑
RAGTruth PRR ↑
RCE ↓
AUROC ↑
PreciseWiki PRR ↑
RCE ↓
AUROC ↑
LongWiki PRR ↑
RCE ↓
AUROC ↑
NonExistent PRR ↑
RCE ↓
Information-based
Maximum Seq. Probability (MSP) Perplexity (PPL) Mean Token Entropy (MTE) Pointwise Mutual Info. (PMI) Conditional PMI Self Certainty Rényi Divergence Fisher–Rao Distance Token SAR Claim-Conditioned Probability
0.76±0.01 0.56±0.02 0.53±0.02 0.35±0.01 0.42±0.01 0.49±0.02 0.46±0.02 0.50±0.02 0.56±0.02 0.76±0.01
0.62±0.03 0.01±0.04 −0.07±0.04 −0.57±0.04 −0.37±0.04 −0.18±0.04 −0.24±0.05 −0.15±0.04 0.00±0.04 0.57±0.03
0.09±0.05 0.49±0.06 0.60±0.06 0.97±0.03 0.95±0.03 0.69±0.08 0.78±0.06 0.65±0.07 0.50±0.06 0.09±0.05
0.76±0.02 0.71±0.02 0.71±0.02 0.47±0.02 0.28±0.02 0.57±0.02 0.55±0.02 0.55±0.02 0.75±0.02 0.74±0.02
0.61±0.05 0.61±0.05 0.61±0.05 −0.09±0.07 −0.82±0.07 0.30±0.06 0.23±0.06 0.23±0.06 0.66±0.05 0.62±0.05
0.19±0.07 0.17±0.06 0.17±0.06 0.91±0.10 1.00±0.00 0.42±0.10 0.49±0.09 0.44±0.11 0.13±0.06 0.20±0.06
0.52±0.02 0.53±0.02 0.52±0.02 0.49±0.02 0.47±0.02 0.52±0.02 0.52±0.02 0.52±0.02 0.53±0.02 0.59±0.02
0.13±0.07 0.14±0.07 0.08±0.07 −0.09±0.09 −0.08±0.08 0.05±0.08 0.04±0.08 0.06±0.08 0.14±0.07 0.35±0.07
0.61±0.12 0.58±0.12 0.61±0.12 0.78±0.10 0.85±0.09 0.62±0.14 0.65±0.13 0.65±0.13 0.57±0.12 0.39±0.09
0.68±0.03 0.55±0.03 0.56±0.03 0.45±0.03 0.44±0.03 0.60±0.03 0.59±0.03 0.61±0.03 0.55±0.03 0.71±0.03
0.44±0.09 0.15±0.10 0.17±0.10 −0.24±0.13 −0.23±0.11 0.24±0.10 0.23±0.11 0.24±0.10 0.13±0.11 0.51±0.09
0.35±0.09 0.50±0.13 0.51±0.13 0.70±0.09 0.94±0.08 0.45±0.10 0.45±0.10 0.45±0.09 0.52±0.13 0.33±0.09
Sampling-based
Monte-Carlo Seq. Entropy Monte-Carlo Norm. Seq. Entropy Semantic Entropy Semantic Density Sentence SAR SAR Cocoa MSP Cocoa PPL Cocoa MTE
0.34±0.01 0.34±0.01 0.35±0.01 0.39±0.01 0.40±0.02 0.35±0.01 0.75±0.01 0.55±0.02 0.54±0.02
−0.56±0.04 −0.57±0.04 −0.52±0.04 −0.41±0.04 −0.44±0.04 −0.48±0.04 0.58±0.03 0.01±0.04 −0.04±0.04
1.00±0.01 1.00±0.00 1.00±0.01 0.98±0.03 0.94±0.04 1.00±0.01 0.08±0.05 0.52±0.08 0.55±0.07
0.73±0.02 0.72±0.02 0.74±0.02 0.78±0.02 0.77±0.02 0.76±0.02 0.77±0.02 0.74±0.02 0.75±0.02
0.58±0.05 0.62±0.05 0.59±0.05 0.72±0.04 0.68±0.05 0.66±0.04 0.66±0.05 0.63±0.05 0.63±0.05
0.19±0.06 0.14±0.06 0.26±0.07 0.12±0.06 0.20±0.07 0.16±0.06 0.21±0.06 0.17±0.07 0.18±0.07
0.52±0.02 0.51±0.02 0.53±0.02 0.57±0.02 0.50±0.02 0.56±0.02 0.56±0.02 0.57±0.02 0.57±0.02
0.01±0.08 0.00±0.08 0.07±0.08 0.22±0.07 0.00±0.08 0.19±0.07 0.22±0.07 0.22±0.07 0.21±0.07
0.66±0.13 0.71±0.12 0.58±0.13 0.42±0.11 0.69±0.10 0.47±0.12 0.49±0.10 0.39±0.11 0.39±0.11
0.66±0.03 0.60±0.03 0.66±0.03 0.43±0.03 0.72±0.03 0.53±0.02 0.62±0.03 0.50±0.03 0.51±0.03
0.44±0.09 0.29±0.09 0.43±0.09 −0.35±0.12 0.54±0.08 0.15±0.10 0.38±0.09 0.04±0.10 0.07±0.10
0.33±0.10 0.44±0.11 0.34±0.10 0.88±0.08 0.33±0.09 0.61±0.13 0.35±0.09 0.72±0.13 0.69±0.13
Internal
Attention Score RAUQ CSL EigenScore
0.73±0.01 0.56±0.02 0.42±0.02 0.32±0.01
0.51±0.04 0.02±0.04 −0.39±0.04 −0.60±0.04
0.23±0.05 0.48±0.06 0.90±0.05 1.00±0.00
0.63±0.02 0.76±0.02 0.69±0.02 0.68±0.02
0.35±0.06 0.64±0.05 0.52±0.05 0.37±0.06
0.31±0.08 0.16±0.07 0.21±0.07 0.41±0.11
0.48±0.02 0.52±0.02 0.49±0.02 0.48±0.02
−0.07±0.08 0.14±0.07 −0.02±0.08 −0.16±0.09
0.77±0.11 0.61±0.12 0.77±0.11 0.80±0.11
0.70±0.03 0.56±0.03 0.48±0.03 0.66±0.03
0.47±0.09 0.18±0.10 −0.05±0.11 0.39±0.10
0.42±0.11 0.51±0.12 0.84±0.11 0.40±0.11
Training-based
Mahalanobis D. Relative Mahalanobis D. RDE HUQ-MD HUQ-RMD
0.24±0.01 0.29±0.01 0.39±0.01 0.56±0.02 0.55±0.02
−0.68±0.04 −0.49±0.04 −0.05±0.04 0.01±0.04 0.00±0.04
1.00±0.00 1.00±0.00 0.94±0.01 0.49±0.06 0.49±0.07
0.53±0.02 0.49±0.02 0.38±0.02 0.71±0.02 0.71±0.02
0.05±0.07 0.02±0.07 −0.51±0.07 0.61±0.05 0.61±0.05
0.60±0.09 0.72±0.10 0.99±0.02 0.17±0.06 0.17±0.06
0.45±0.02 0.49±0.02 0.58±0.02 0.53±0.02 0.53±0.02
−0.15±0.08 −0.04±0.08 0.27±0.07 0.14±0.07 0.14±0.07
0.92±0.08 0.81±0.12 0.37±0.11 0.57±0.12 0.57±0.12
0.26±0.02 0.21±0.02 0.32±0.02 0.55±0.03 0.55±0.03
−1.01±0.12 −1.31±0.11 −0.61±0.11 0.15±0.10 0.15±0.10
1.00±0.00 1.00±0.00 1.00±0.00 0.50±0.13 0.50±0.13
Reflexive
P(True) P(True) Sampling P(True) Empirical
0.48±0.01 0.37±0.01 0.51±0.00
−0.19±0.04 −0.34±0.04 0.01±0.04
0.73±0.06 1.00±0.01 0.69±0.15
0.48±0.02 0.54±0.02 0.51±0.01
0.04±0.07 0.19±0.06 0.01±0.07
0.79±0.09 0.49±0.12 0.69±0.16
0.48±0.02 – 0.50±0.00
−0.12±0.08 – 0.00±0.08
0.83±0.11 – 0.75±0.15
0.26±0.03 – 0.49±0.00
−1.19±0.13 – −0.02±0.11
0.84±0.11 – 0.76±0.15
Black-box
NumSet LabelProb Kernel Language Entropy (KLE) EigValLap NLI (Entail) EigValLap NLI (Contra) EigValLap Jaccard Eccentricity NLI (Entail) Eccentricity NLI (Contra) Eccentricity Jaccard DegMat NLI (Entail) DegMat NLI (Contra) DegMat Jaccard LUQ Lexical Sim. (ROUGE-L) Lexical Sim. (BLEU)
0.46±0.01 0.34±0.01 0.33±0.01 0.33±0.01 0.39±0.01 0.30±0.01 0.33±0.01 0.40±0.01 0.48±0.01 0.33±0.01 0.37±0.01 0.30±0.01 0.35±0.01 0.29±0.01 0.31±0.01
−0.10±0.04 −0.48±0.04 −0.50±0.04 −0.50±0.04 −0.25±0.04 −0.66±0.04 −0.51±0.04 −0.25±0.04 −0.04±0.04 −0.51±0.04 −0.33±0.04 −0.65±0.04 −0.42±0.04 −0.65±0.04 −0.61±0.04
0.92±0.08 1.00±0.00 1.00±0.00 1.00±0.00 1.00±0.00 1.00±0.00 1.00±0.00 1.00±0.00 0.84±0.12 1.00±0.00 1.00±0.00 1.00±0.00 1.00±0.00 1.00±0.00 1.00±0.00
0.78±0.01 0.76±0.02 0.79±0.02 0.80±0.01 0.76±0.02 0.68±0.02 0.77±0.02 0.79±0.01 0.72±0.02 0.80±0.01 0.76±0.02 0.70±0.02 0.79±0.01 0.70±0.02 0.69±0.02
0.75±0.04 0.64±0.05 0.74±0.04 0.76±0.04 0.70±0.05 0.54±0.05 0.63±0.05 0.75±0.04 0.56±0.05 0.75±0.04 0.71±0.04 0.52±0.05 0.75±0.04 0.56±0.05 0.51±0.05
0.11±0.05 0.17±0.06 0.11±0.06 0.15±0.06 0.15±0.06 0.18±0.07 0.25±0.07 0.13±0.05 0.21±0.07 0.13±0.06 0.12±0.05 0.19±0.07 0.12±0.06 0.19±0.07 0.19±0.07
0.53±0.01 0.52±0.02 0.58±0.02 0.56±0.02 0.61±0.02 0.55±0.02 0.55±0.02 0.62±0.02 0.50±0.02 0.55±0.02 0.61±0.02 0.55±0.02 0.61±0.02 0.53±0.02 0.54±0.02
0.13±0.08 0.05±0.08 0.27±0.07 0.22±0.07 0.35±0.07 0.18±0.07 0.11±0.08 0.32±0.07 0.02±0.08 0.18±0.08 0.31±0.07 0.19±0.07 0.32±0.07 0.14±0.08 0.12±0.08
0.58±0.14 0.63±0.14 0.39±0.11 0.41±0.11 0.36±0.09 0.48±0.11 0.50±0.13 0.30±0.09 0.71±0.15 0.45±0.12 0.32±0.09 0.49±0.11 0.28±0.09 0.58±0.13 0.53±0.12
0.44±0.02 0.59±0.02 0.50±0.03 0.53±0.03 0.47±0.03 0.43±0.03 0.57±0.03 0.46±0.03 0.51±0.02 0.55±0.03 0.46±0.03 0.44±0.03 0.46±0.03 0.39±0.02 0.34±0.02
−0.29±0.11 0.21±0.10 −0.05±0.11 0.13±0.10 −0.10±0.10 −0.27±0.12 0.12±0.11 −0.17±0.10 0.04±0.11 0.20±0.10 −0.12±0.10 −0.25±0.12 −0.14±0.11 −0.36±0.11 −0.57±0.11
0.95±0.07 0.44±0.11 0.74±0.11 0.65±0.11 0.86±0.10 0.91±0.08 0.53±0.11 0.91±0.09 0.68±0.15 0.58±0.11 0.88±0.11 0.89±0.09 0.83±0.11 0.99±0.03 1.00±0.00
AlignScore
Table 4: AUROC ↑, PRR ↑, RCE ↓ for Mistral-7B-Instruct across tasks (fixed metric-wise gradient; warmer = higher value). Uncertainty estimators and AlignScore variants are ranked separately: bold = best per column (ties marked); underline = second-best (ties marked) within the uncertainty group; AlignScore group shows only the best. Uncertainty Metric
AS-Precision(gold_answer) AS-Precision(wikipage) AS-Recall(gold_answer) AS-Recall(wikipage)
0.66±0.01 – 0.58±0.01 –
0.22±0.04 – 0.04±0.04 –
0.42±0.05 – 0.56±0.06 –
0.73±0.02 0.85±0.01 0.91±0.01 0.49±0.02
0.34±0.06 0.85±0.03 0.81±0.04 −0.05±0.07
0.67±0.08 0.13±0.05 0.43±0.11 0.79±0.13
0.61±0.02 0.60±0.02 0.59±0.02 0.56±0.02
0.34±0.07 0.40±0.06 0.36±0.07 0.22±0.07
0.35±0.08 0.36±0.08 0.37±0.10 0.49±0.12
0.71±0.03 – 0.72±0.03 –
0.49±0.09 – 0.52±0.08 –
0.33±0.09 – 0.39±0.10 –
29
AUROC ↑
RAGTruth PRR ↑
RCE ↓
AUROC ↑
PreciseWiki PRR ↑
RCE ↓
AUROC ↑
LongWiki PRR ↑
RCE ↓
AUROC ↑
NonExistent PRR ↑
RCE ↓
Information-based
Maximum Seq. Probability (MSP) Perplexity (PPL) Mean Token Entropy (MTE) Pointwise Mutual Info. (PMI) Conditional PMI Self Certainty Rényi Divergence Fisher–Rao Distance Token SAR Claim-Conditioned Probability
0.53±0.01 0.41±0.01 0.53±0.01 0.32±0.01 0.55±0.01 0.46±0.01 0.46±0.01 0.46±0.01 0.41±0.01 0.49±0.01
−0.01±0.03 −0.28±0.04 −0.02±0.04 −0.46±0.04 0.06±0.03 −0.20±0.04 −0.21±0.04 −0.21±0.04 −0.28±0.04 −0.03±0.03
0.65±0.08 0.99±0.02 0.59±0.09 1.00±0.00 0.53±0.10 0.87±0.05 0.93±0.05 0.85±0.05 0.99±0.02 0.80±0.07
0.74±0.02 0.68±0.02 0.64±0.02 0.46±0.03 0.29±0.02 0.59±0.02 0.58±0.02 0.58±0.02 0.74±0.02 0.72±0.02
0.62±0.07 0.66±0.06 0.61±0.06 −0.22±0.11 −0.84±0.11 0.52±0.07 0.44±0.07 0.46±0.07 0.74±0.05 0.61±0.08
0.30±0.07 0.21±0.07 0.32±0.09 0.88±0.10 1.00±0.01 0.49±0.08 0.52±0.08 0.54±0.08 0.16±0.06 0.26±0.07
0.56±0.01 0.59±0.01 0.59±0.01 0.53±0.01 0.41±0.01 0.57±0.01 0.56±0.01 0.57±0.01 0.59±0.01 0.63±0.01
0.22±0.04 0.27±0.04 0.28±0.04 0.12±0.05 −0.23±0.05 0.20±0.04 0.18±0.04 0.23±0.04 0.25±0.04 0.39±0.04
0.42±0.08 0.24±0.08 0.30±0.08 0.51±0.12 1.00±0.01 0.29±0.09 0.32±0.10 0.28±0.09 0.23±0.08 0.22±0.07
0.57±0.04 0.49±0.04 0.51±0.04 0.46±0.03 0.51±0.04 0.52±0.04 0.51±0.04 0.54±0.04 0.50±0.04 0.64±0.04
0.13±0.14 −0.11±0.16 −0.01±0.16 −0.02±0.14 −0.06±0.15 −0.01±0.16 −0.07±0.16 0.04±0.16 −0.10±0.16 0.36±0.13
0.61±0.12 0.73±0.12 0.72±0.13 0.78±0.08 0.70±0.12 0.64±0.11 0.59±0.11 0.57±0.09 0.69±0.12 0.42±0.11
Sampling-based
Monte-Carlo Seq. Entropy Monte-Carlo Norm. Seq. Entropy Semantic Entropy Semantic Density Sentence SAR SAR Cocoa MSP Cocoa PPL Cocoa MTE
0.46±0.01 0.38±0.01 0.46±0.01 0.48±0.01 0.56±0.01 0.42±0.01 0.50±0.01 0.42±0.01 0.50±0.01
−0.26±0.03 −0.37±0.03 −0.25±0.03 −0.07±0.04 −0.02±0.04 −0.28±0.04 −0.01±0.03 −0.27±0.04 −0.07±0.04
0.82±0.06 1.00±0.00 0.84±0.05 0.84±0.11 0.57±0.05 0.97±0.04 0.76±0.10 0.99±0.03 0.72±0.08
0.71±0.02 0.70±0.02 0.73±0.02 0.71±0.03 0.75±0.02 0.75±0.02 0.76±0.02 0.72±0.02 0.70±0.02
0.63±0.07 0.63±0.07 0.63±0.07 0.48±0.08 0.64±0.07 0.66±0.07 0.66±0.07 0.70±0.06 0.71±0.06
0.19±0.07 0.21±0.07 0.22±0.08 0.45±0.11 0.27±0.08 0.27±0.07 0.27±0.08 0.17±0.07 0.16±0.07
0.51±0.01 0.54±0.01 0.51±0.01 0.58±0.01 0.49±0.01 0.61±0.02 0.63±0.01 0.65±0.01 0.65±0.01
0.03±0.05 0.06±0.05 0.04±0.05 0.23±0.05 0.00±0.05 0.22±0.04 0.35±0.04 0.39±0.04 0.39±0.04
0.65±0.11 0.53±0.11 0.65±0.15 0.28±0.09 0.79±0.12 0.35±0.08 0.18±0.07 0.16±0.06 0.15±0.06
0.56±0.04 0.44±0.04 0.55±0.04 0.49±0.03 0.64±0.03 0.39±0.03 0.49±0.04 0.44±0.04 0.45±0.04
0.16±0.14 −0.29±0.16 0.15±0.14 −0.04±0.14 0.52±0.10 −0.42±0.16 −0.14±0.16 −0.31±0.16 −0.20±0.16
0.65±0.10 0.90±0.10 0.63±0.11 0.79±0.12 0.32±0.10 0.95±0.06 0.78±0.10 0.87±0.09 0.86±0.10
Internal
Attention Score RAUQ CSL EigenScore
0.65±0.01 0.42±0.01 0.37±0.01 0.48±0.01
0.30±0.03 −0.27±0.04 −0.34±0.03 −0.17±0.03
0.30±0.06 0.98±0.03 1.00±0.00 0.83±0.07
0.56±0.02 0.73±0.02 0.59±0.02 0.64±0.03
0.30±0.10 0.66±0.06 0.38±0.09 0.29±0.10
0.55±0.11 0.20±0.07 0.39±0.11 0.48±0.11
0.45±0.01 0.58±0.01 0.56±0.02 0.42±0.01
−0.16±0.05 0.25±0.04 0.18±0.05 −0.21±0.05
0.95±0.06 0.28±0.08 0.36±0.10 0.99±0.02
0.60±0.04 0.50±0.04 0.45±0.03 0.59±0.04
0.24±0.13 −0.10±0.16 −0.07±0.14 0.13±0.15
0.50±0.13 0.72±0.11 0.85±0.10 0.62±0.10
Training-based
Mahalanobis D. Relative Mahalanobis D. RDE HUQ-MD HUQ-RMD
0.33±0.01 0.33±0.01 0.33±0.01 0.41±0.01 0.41±0.01
−0.31±0.04 −0.32±0.04 −0.28±0.04 −0.28±0.04 −0.28±0.04
1.00±0.00 1.00±0.00 0.99±0.02 0.99±0.02 0.99±0.02
0.52±0.03 0.52±0.03 0.55±0.03 0.68±0.02 0.68±0.02
0.05±0.10 0.04±0.11 0.21±0.09 0.66±0.06 0.66±0.06
0.65±0.15 0.66±0.15 0.53±0.12 0.19±0.07 0.19±0.07
0.40±0.01 0.42±0.01 0.44±0.01 0.59±0.01 0.59±0.01
−0.28±0.05 −0.23±0.05 −0.20±0.05 0.27±0.04 0.27±0.04
1.00±0.01 0.99±0.03 1.00±0.02 0.24±0.08 0.24±0.08
0.18±0.03 0.18±0.03 0.25±0.02 0.49±0.04 0.49±0.04
−1.49±0.13 −1.49±0.14 −1.06±0.12 −0.11±0.16 −0.11±0.16
1.00±0.01 1.00±0.01 1.00±0.00 0.73±0.12 0.73±0.12
Reflexive
P(True) P(True) Sampling P(True) Empirical
0.43±0.01 0.50±0.01 0.52±0.01
−0.19±0.04 −0.02±0.04 0.08±0.03
0.98±0.03 0.72±0.12 0.53±0.11
0.58±0.03 0.53±0.03 0.44±0.02
0.31±0.10 0.09±0.10 −0.11±0.10
0.44±0.12 0.62±0.13 0.87±0.09
0.49±0.02 – 0.49±0.02
−0.06±0.05 – −0.08±0.05
0.83±0.10 – 0.85±0.12
0.39±0.04 – 0.54±0.03
−0.54±0.17 – 0.10±0.14
0.93±0.07 – 0.60±0.15
Black-box
NumSet LabelProb Kernel Language Entropy (KLE) EigValLap NLI (Entail) EigValLap NLI (Contra) EigValLap Jaccard Eccentricity NLI (Entail) Eccentricity NLI (Contra) Eccentricity Jaccard DegMat NLI (Entail) DegMat NLI (Contra) DegMat Jaccard LUQ Lexical Sim. (ROUGE-L) Lexical Sim. (BLEU)
0.48±0.01 0.52±0.01 0.47±0.01 0.47±0.01 0.43±0.01 0.37±0.01 0.51±0.01 0.43±0.01 0.56±0.01 0.48±0.01 0.41±0.01 0.40±0.01 0.37±0.01 0.40±0.01 0.42±0.01
−0.08±0.04 0.01±0.04 −0.14±0.04 −0.14±0.04 −0.09±0.03 −0.38±0.04 −0.02±0.04 −0.11±0.04 0.09±0.03 −0.12±0.04 −0.15±0.04 −0.34±0.04 −0.30±0.03 −0.34±0.04 −0.31±0.04
0.81±0.12 0.58±0.11 0.85±0.07 0.83±0.10 0.95±0.05 1.00±0.00 0.68±0.12 0.96±0.05 0.46±0.11 0.78±0.09 0.99±0.03 0.97±0.03 1.00±0.01 0.96±0.04 0.95±0.04
0.73±0.02 0.71±0.02 0.70±0.03 0.72±0.03 0.60±0.03 0.65±0.03 0.72±0.03 0.70±0.02 0.67±0.03 0.73±0.03 0.62±0.03 0.67±0.03 0.71±0.03 0.64±0.03 0.66±0.03
0.60±0.07 0.53±0.08 0.45±0.09 0.52±0.08 0.22±0.09 0.40±0.09 0.53±0.09 0.54±0.08 0.37±0.09 0.54±0.08 0.26±0.10 0.43±0.09 0.49±0.08 0.38±0.10 0.45±0.09
0.22±0.08 0.28±0.09 0.41±0.11 0.34±0.09 0.55±0.11 0.38±0.09 0.33±0.09 0.30±0.09 0.40±0.09 0.37±0.10 0.49±0.12 0.39±0.09 0.38±0.11 0.40±0.11 0.31±0.09
0.51±0.01 0.52±0.01 0.58±0.02 0.57±0.02 0.65±0.01 0.60±0.01 0.57±0.02 0.64±0.01 0.48±0.01 0.55±0.02 0.65±0.01 0.61±0.01 0.65±0.01 0.56±0.01 0.60±0.01
0.05±0.05 0.06±0.05 0.20±0.05 0.16±0.05 0.40±0.04 0.25±0.04 0.14±0.05 0.36±0.04 −0.06±0.05 0.12±0.05 0.38±0.04 0.28±0.04 0.35±0.04 0.17±0.05 0.26±0.04
0.62±0.14 0.57±0.14 0.29±0.08 0.38±0.09 0.15±0.06 0.28±0.08 0.41±0.10 0.17±0.07 0.85±0.12 0.45±0.11 0.15±0.06 0.24±0.08 0.21±0.08 0.35±0.10 0.21±0.07
0.45±0.03 0.49±0.03 0.50±0.04 0.47±0.04 0.53±0.04 0.32±0.03 0.49±0.03 0.54±0.04 0.52±0.03 0.48±0.04 0.53±0.04 0.33±0.03 0.49±0.04 0.25±0.03 0.25±0.03
−0.23±0.16 −0.05±0.15 −0.08±0.16 −0.13±0.16 0.11±0.15 −0.70±0.15 0.04±0.15 0.17±0.15 0.08±0.14 −0.11±0.16 0.05±0.15 −0.69±0.15 −0.16±0.16 −1.10±0.14 −1.17±0.14
0.84±0.12 0.77±0.13 0.74±0.10 0.80±0.12 0.66±0.13 1.00±0.01 0.75±0.11 0.61±0.12 0.67±0.15 0.80±0.13 0.65±0.15 1.00±0.02 0.75±0.11 1.00±0.00 1.00±0.01
AlignScore
Table 5: AUROC ↑, PRR ↑, RCE ↓ for Llama-2-7B-Chat across tasks (fixed metric-wise gradient; warmer = higher value). Uncertainty estimators and AlignScore variants are ranked separately: bold = best per column (ties marked); underline = second-best (ties marked) within the uncertainty group; AlignScore group shows only the best. Uncertainty Metric
AS-Precision(gold_answer) AS-Precision(wikipage) AS-Recall(gold_answer) AS-Recall(wikipage)
0.71±0.01 – 0.57±0.01 –
0.42±0.03 – 0.10±0.03 –
0.11±0.05 – 0.43±0.09 –
0.88±0.02 0.78±0.02 0.90±0.02 0.56±0.02
0.82±0.06 0.84±0.05 0.80±0.06 0.27±0.09
0.41±0.12 0.18±0.07 0.66±0.10 0.52±0.12
0.65±0.01 0.70±0.01 0.58±0.01 0.57±0.01
0.40±0.04 0.52±0.04 0.19±0.04 0.17±0.04
0.17±0.06 0.10±0.05 0.31±0.09 0.30±0.09
0.55±0.04 – 0.61±0.04 –
0.12±0.14 – 0.16±0.14 –
0.63±0.11 – 0.64±0.12 –
30
AUROC ↑
RAGTruth PRR ↑
RCE ↓
AUROC ↑
PreciseWiki PRR ↑
RCE ↓
AUROC ↑
LongWiki PRR ↑
RCE ↓
AUROC ↑
NonExistent PRR ↑
RCE ↓
Information-based
Maximum Seq. Probability (MSP) Perplexity (PPL) Mean Token Entropy (MTE) Pointwise Mutual Info. (PMI) Conditional PMI Self Certainty Rényi Divergence Fisher–Rao Distance Token SAR Claim-Conditioned Probability
0.63±0.01 0.35±0.01 0.51±0.01 0.31±0.01 0.56±0.01 0.43±0.01 0.42±0.01 0.44±0.01 0.35±0.01 0.58±0.01
0.23±0.03 −0.40±0.03 −0.15±0.03 −0.46±0.03 0.04±0.03 −0.30±0.03 −0.31±0.03 −0.29±0.03 −0.40±0.03 0.13±0.03
0.24±0.05 1.00±0.00 0.75±0.04 1.00±0.01 0.50±0.07 0.91±0.05 0.95±0.04 0.91±0.04 1.00±0.00 0.31±0.10
0.64±0.02 0.55±0.02 0.56±0.02 0.47±0.02 0.34±0.02 0.54±0.02 0.53±0.02 0.51±0.02 0.63±0.02 0.70±0.02
0.44±0.06 0.09±0.07 0.14±0.07 −0.18±0.08 −0.54±0.08 0.20±0.07 0.17±0.07 0.10±0.07 0.30±0.07 0.61±0.05
0.34±0.07 0.60±0.09 0.57±0.09 0.87±0.09 0.96±0.04 0.54±0.09 0.62±0.11 0.66±0.09 0.36±0.10 0.17±0.06
0.59±0.02 0.59±0.02 0.59±0.02 0.49±0.02 0.40±0.02 0.55±0.02 0.54±0.02 0.56±0.02 0.60±0.02 0.64±0.02
0.28±0.06 0.28±0.06 0.27±0.06 −0.04±0.07 −0.35±0.07 0.13±0.07 0.10±0.07 0.16±0.07 0.31±0.06 0.46±0.06
0.30±0.09 0.29±0.09 0.29±0.09 0.76±0.12 0.98±0.04 0.50±0.11 0.51±0.12 0.47±0.10 0.29±0.09 0.23±0.07
0.76±0.02 0.25±0.02 0.26±0.02 0.42±0.02 0.75±0.02 0.40±0.02 0.41±0.02 0.45±0.02 0.21±0.02 0.85±0.01
0.58±0.05 −1.00±0.07 −0.93±0.07 −0.07±0.07 0.60±0.05 −0.45±0.07 −0.42±0.07 −0.34±0.07 −1.19±0.07 0.79±0.04
0.18±0.07 1.00±0.00 1.00±0.00 0.85±0.05 0.19±0.06 0.75±0.11 0.72±0.09 0.59±0.07 1.00±0.01 0.16±0.07
Sampling-based
Monte-Carlo Seq. Entropy Monte-Carlo Norm. Seq. Entropy Semantic Entropy Semantic Density Sentence SAR SAR Cocoa MSP Cocoa PPL Cocoa MTE
0.46±0.01 0.38±0.01 0.49±0.01 0.49±0.01 0.58±0.01 0.46±0.01 0.60±0.01 0.38±0.01 0.48±0.01
−0.26±0.03 −0.33±0.03 −0.16±0.04 −0.05±0.03 0.04±0.03 −0.19±0.03 0.15±0.03 −0.37±0.03 −0.19±0.03
0.80±0.05 0.99±0.02 0.77±0.04 0.78±0.11 0.53±0.04 0.82±0.06 0.34±0.09 1.00±0.00 0.78±0.05
0.58±0.02 0.58±0.02 0.59±0.02 0.75±0.02 0.63±0.02 0.65±0.02 0.69±0.02 0.60±0.02 0.61±0.02
0.22±0.07 0.24±0.07 0.27±0.07 0.64±0.05 0.45±0.06 0.41±0.06 0.58±0.05 0.31±0.06 0.36±0.06
0.42±0.12 0.47±0.10 0.39±0.11 0.14±0.06 0.34±0.06 0.28±0.08 0.22±0.06 0.43±0.08 0.35±0.09
0.54±0.02 0.55±0.02 0.58±0.02 0.62±0.02 0.54±0.02 0.61±0.02 0.64±0.02 0.65±0.02 0.64±0.02
0.12±0.07 0.13±0.07 0.24±0.06 0.42±0.06 0.14±0.07 0.29±0.06 0.43±0.06 0.44±0.06 0.42±0.06
0.52±0.12 0.51±0.12 0.40±0.10 0.27±0.08 0.53±0.12 0.33±0.10 0.21±0.07 0.21±0.07 0.18±0.07
0.52±0.02 0.40±0.02 0.51±0.02 0.33±0.02 0.61±0.02 0.36±0.02 0.68±0.02 0.23±0.02 0.24±0.02
0.10±0.06 −0.28±0.06 0.06±0.06 −0.58±0.06 0.34±0.05 −0.42±0.07 0.42±0.05 −1.09±0.06 −1.02±0.07
0.58±0.13 0.99±0.02 0.68±0.13 1.00±0.00 0.28±0.08 1.00±0.01 0.27±0.09 1.00±0.00 1.00±0.00
Internal
Attention Score RAUQ CSL EigenScore
0.69±0.01 0.35±0.01 0.38±0.01 0.51±0.01
0.41±0.03 −0.40±0.03 −0.38±0.03 −0.10±0.03
0.30±0.05 1.00±0.00 0.99±0.01 0.78±0.04
0.58±0.02 0.56±0.02 0.60±0.02 0.53±0.02
0.29±0.06 0.14±0.07 0.29±0.06 0.03±0.07
0.39±0.10 0.67±0.09 0.44±0.08 0.62±0.11
0.47±0.02 0.58±0.02 0.54±0.02 0.50±0.02
−0.13±0.07 0.29±0.06 0.13±0.07 0.08±0.07
0.90±0.10 0.32±0.10 0.50±0.12 0.72±0.12
0.87±0.01 0.27±0.02 0.49±0.02 0.42±0.02
0.74±0.04 −0.93±0.07 −0.35±0.07 −0.20±0.06
0.35±0.07 1.00±0.00 0.79±0.08 0.98±0.04
Training-based
Mahalanobis D. Relative Mahalanobis D. RDE HUQ-MD HUQ-RMD
0.23±0.01 0.25±0.01 0.25±0.01 0.35±0.01 0.35±0.01
−0.44±0.03 −0.49±0.03 −0.37±0.03 −0.40±0.03 −0.40±0.03
1.00±0.00 1.00±0.00 1.00±0.01 1.00±0.00 1.00±0.00
0.47±0.02 0.46±0.02 0.59±0.02 0.55±0.02 0.55±0.02
−0.01±0.07 −0.05±0.07 0.37±0.06 0.07±0.07 0.07±0.07
0.78±0.10 0.84±0.10 0.33±0.09 0.62±0.09 0.62±0.09
0.35±0.02 0.36±0.02 0.46±0.02 0.59±0.02 0.59±0.02
−0.53±0.08 −0.49±0.08 −0.12±0.07 0.28±0.06 0.28±0.06
1.00±0.00 1.00±0.00 0.92±0.08 0.29±0.09 0.29±0.09
0.60±0.02 0.52±0.02 0.52±0.02 0.25±0.02 0.25±0.02
0.19±0.07 0.06±0.06 0.18±0.05 −1.00±0.07 −1.00±0.07
0.45±0.07 0.65±0.10 0.69±0.05 1.00±0.00 1.00±0.00
Reflexive
P(True) P(True) Sampling P(True) Empirical
0.63±0.01 0.63±0.01 0.67±0.01
0.25±0.03 0.26±0.03 0.40±0.03
0.19±0.07 0.19±0.07 0.13±0.06
0.46±0.02 0.46±0.02 0.52±0.02
−0.05±0.07 −0.16±0.08 0.04±0.07
0.87±0.10 0.78±0.10 0.61±0.15
0.45±0.02 – 0.49±0.02
−0.15±0.07 – −0.05±0.07
0.95±0.07 – 0.80±0.14
0.61±0.02 – 0.33±0.02
0.44±0.06 – −0.60±0.06
0.52±0.05 – 1.00±0.00
Black-box
NumSet LabelProb Kernel Language Entropy (KLE) EigValLap NLI (Entail) EigValLap NLI (Contra) EigValLap Jaccard Eccentricity NLI (Entail) Eccentricity NLI (Contra) Eccentricity Jaccard DegMat NLI (Entail) DegMat NLI (Contra) DegMat Jaccard LUQ Lexical Sim. (ROUGE-L) Lexical Sim. (BLEU)
0.47±0.01 0.51±0.01 0.49±0.01 0.51±0.01 0.43±0.01 0.42±0.01 0.52±0.01 0.42±0.01 0.57±0.01 0.51±0.01 0.41±0.01 0.45±0.01 0.43±0.01 0.43±0.01 0.46±0.01
−0.09±0.03 0.00±0.03 −0.04±0.03 0.03±0.03 −0.16±0.03 −0.27±0.03 0.04±0.03 −0.17±0.03 0.09±0.03 0.02±0.03 −0.20±0.03 −0.21±0.03 −0.15±0.03 −0.29±0.03 −0.21±0.03
0.81±0.13 0.62±0.11 0.71±0.11 0.66±0.13 0.98±0.04 0.95±0.04 0.59±0.10 0.99±0.03 0.57±0.11 0.63±0.13 0.99±0.02 0.85±0.05 0.93±0.06 0.94±0.04 0.86±0.06
0.70±0.02 0.66±0.02 0.72±0.02 0.71±0.02 0.66±0.02 0.63±0.02 0.67±0.02 0.71±0.02 0.61±0.02 0.73±0.02 0.67±0.02 0.63±0.02 0.73±0.02 0.65±0.02 0.61±0.02
0.57±0.05 0.40±0.06 0.62±0.05 0.64±0.05 0.41±0.06 0.35±0.06 0.40±0.07 0.58±0.05 0.29±0.07 0.64±0.05 0.46±0.06 0.36±0.06 0.63±0.05 0.43±0.06 0.32±0.06
0.13±0.06 0.28±0.08 0.18±0.07 0.15±0.06 0.25±0.09 0.30±0.09 0.31±0.08 0.18±0.07 0.29±0.10 0.13±0.06 0.30±0.07 0.24±0.09 0.17±0.07 0.21±0.08 0.32±0.10
0.55±0.01 0.59±0.02 0.64±0.02 0.62±0.02 0.68±0.02 0.60±0.02 0.62±0.02 0.67±0.02 0.49±0.02 0.62±0.02 0.68±0.02 0.60±0.02 0.68±0.02 0.55±0.02 0.58±0.02
0.21±0.06 0.27±0.06 0.42±0.06 0.35±0.06 0.50±0.06 0.31±0.06 0.34±0.06 0.44±0.06 −0.03±0.07 0.34±0.06 0.51±0.05 0.32±0.06 0.51±0.05 0.16±0.06 0.25±0.06
0.55±0.13 0.32±0.10 0.21±0.07 0.23±0.08 0.17±0.07 0.30±0.09 0.29±0.08 0.23±0.07 0.78±0.15 0.24±0.08 0.19±0.07 0.29±0.09 0.21±0.07 0.45±0.11 0.36±0.10
0.45±0.01 0.45±0.02 0.40±0.02 0.40±0.02 0.44±0.02 0.40±0.02 0.44±0.02 0.44±0.02 0.53±0.01 0.40±0.02 0.44±0.02 0.40±0.02 0.40±0.02 0.34±0.02 0.32±0.02
−0.21±0.06 −0.14±0.06 −0.28±0.06 −0.28±0.06 −0.16±0.06 −0.32±0.06 −0.13±0.06 −0.17±0.06 0.11±0.06 −0.28±0.06 −0.19±0.06 −0.34±0.07 −0.32±0.06 −0.53±0.07 −0.62±0.06
0.93±0.08 0.96±0.06 0.98±0.03 0.99±0.02 0.97±0.05 1.00±0.01 0.93±0.07 0.97±0.05 0.53±0.14 0.99±0.02 0.96±0.06 1.00±0.01 0.98±0.04 1.00±0.00 1.00±0.00
AlignScore
Table 6: AUROC ↑, PRR ↑, RCE ↓ for Llama-2-13B-Chat across tasks (fixed metric-wise gradient; warmer = higher value). Uncertainty estimators and AlignScore variants are ranked separately: bold = best per column (ties marked); underline = second-best (ties marked) within the uncertainty group; AlignScore group shows only the best. Uncertainty Metric
AS-Precision(gold_answer) AS-Precision(wikipage) AS-Recall(gold_answer) AS-Recall(wikipage)
0.74±0.01 – 0.51±0.01 –
0.47±0.03 – −0.11±0.03 –
1.00±0.06 – 0.73±0.06 –
0.84±0.02 0.80±0.01 0.90±0.01 0.50±0.02
0.65±0.05 0.81±0.04 0.80±0.04 −0.04±0.08
0.54±0.09 0.10±0.05 0.39±0.10 0.73±0.11
0.69±0.02 0.72±0.02 0.61±0.02 0.57±0.02
0.56±0.05 0.58±0.05 0.30±0.06 0.20±0.07
0.17±0.07 0.17±0.06 0.31±0.08 0.39±0.10
0.74±0.02 – 0.62±0.02 –
0.54±0.05 – −0.02±0.07 –
0.30±0.06 – 0.82±0.05 –
31
Table 7: Mean estimator ranks across benchmark panels, pooling all tasks and models. Within each panel, estimators are ranked by the corresponding metric, with rank 1 indicating the best score. Each cell reports the mean rank, with the subscript indicating one standard deviation across panels; lower is better. Parentheses indicate the number of panels in which the estimator ranks in the top 3. Bold marks the best mean rank within uncertainty estimators and within AlignScore variants separately; underline marks the second-best uncertainty estimator. Estimator
AUROC
PRR
RCE
Maximum Seq. Probability (MSP) Perplexity (PPL) Mean Token Entropy (MTE) Pointwise Mutual Info. (PMI) Conditional PMI Self Certainty Rényi Divergence Fisher–Rao Distance Token SAR Claim-Conditioned Probability (CCP) Monte-Carlo Seq. Entropy Monte-Carlo Norm. Seq. Entropy Semantic Entropy Semantic Density Sentence SAR SAR Cocoa MSP Cocoa PPL Cocoa MTE Attention Score RAUQ CSL EigenScore Mahalanobis Distance (MD) Relative MD (RMD) Robust Density Estimation (RDE) HUQ-MD HUQ-RMD P(True) P(True) Sampling P(True) Empirical NumSet LabelProb Kernel Language Entropy (KLE) EigValLap NLI (entail) EigValLap NLI (contra) EigValLap Jaccard Eccentricity NLI (entail) Eccentricity NLI (contra) Eccentricity Jaccard DegMat NLI (entail) DegMat NLI (contra) DegMat Jaccard LUQ Lexical Sim. (ROUGE-L) Lexical Sim. (BLEU)
13.3±10.3 (2) 27.4±9.9 (0) 23.7±9.8 (0) 40.1±7.4 (0) 32.6±18.1 (0) 28.1±8.9 (0) 30.4±9.0 (0) 26.8±11.0 (0) 24.1±11.7 (0) 8.9±6.1 (4) 24.8±10.7 (0) 31.8±7.8 (0) 22.1±9.6 (0) 20.2±10.9 (0) 17.2±14.5 (2) 22.2±9.9 (0) 9.9±5.1 (1) 23.3±13.2 (0) 20.6±11.2 (1) 21.9±19.3 (3) 24.8±10.7 (0) 32.8±7.5 (0) 29.5±14.3 (0) 43.8±10.4 (0) 44.0±9.1 (0) 36.8±13.2 (0) 28.0±9.9 (0) 28.3±9.3 (0) 34.8±15.2 (0) 30.5±15.9 (0) 30.4±15.4 (1) 23.8±10.2 (0) 22.2±8.9 (0) 19.8±9.5 (0) 20.3±10.6 (1) 18.4±11.4 (2) 31.2±10.1 (0) 18.8±7.8 (0) 17.2±11.3 (1) 24.2±12.4 (0) 20.2±11.7 (0) 20.8±12.6 (1) 29.4±9.6 (0) 20.5±13.7 (0) 33.7±8.1 (0) 31.5±9.0 (0)
13.8±9.0 (1) 26.2±11.3 (0) 23.1±9.1 (0) 39.8±10.7 (0) 32.7±17.7 (1) 28.5±7.8 (0) 30.8±7.3 (0) 27.6±8.4 (0) 24.2±12.1 (0) 7.8±6.4 (5) 26.2±12.4 (0) 31.6±10.0 (0) 24.5±10.5 (0) 20.7±10.6 (0) 17.8±13.6 (2) 24.8±8.4 (0) 11.3±7.5 (1) 21.9±14.0 (0) 19.8±11.4 (0) 21.2±18.5 (4) 23.9±10.7 (0) 33.2±5.7 (0) 31.3±13.6 (0) 42.7±10.4 (0) 42.1±9.6 (0) 33.2±14.3 (0) 26.9±11.6 (0) 26.9±11.1 (0) 33.5±14.9 (0) 28.8±17.4 (0) 32.0±17.1 (1) 23.5±10.6 (0) 22.8±7.0 (0) 20.2±9.3 (0) 20.1±10.3 (1) 17.2±11.3 (1) 31.7±10.2 (0) 20.7±8.4 (0) 15.3±9.1 (0) 24.7±14.2 (1) 20.3±10.3 (0) 18.5±11.2 (1) 30.2±10.7 (0) 20.6±12.7 (0) 34.5±8.6 (0) 33.6±8.3 (0)
14.3±10.0 (2) 22.7±12.7 (0) 20.4±9.3 (0) 36.8±9.3 (0) 32.5±18.8 (2) 24.0±10.8 (0) 26.6±12.7 (0) 25.6±11.7 (0) 20.6±13.2 (1) 10.3±7.5 (4) 20.7±12.6 (1) 30.8±12.5 (0) 19.9±9.8 (1) 22.9±12.4 (1) 18.5±13.5 (1) 25.8±9.7 (0) 14.3±9.6 (1) 23.0±13.3 (0) 17.0±12.5 (2) 22.2±18.4 (2) 24.3±13.4 (1) 31.6±8.3 (0) 31.0±12.0 (0) 40.6±9.9 (0) 40.5±9.4 (0) 33.6±13.1 (0) 22.7±13.1 (0) 22.9±12.8 (0) 33.0±14.8 (1) 29.2±17.3 (1) 30.9±15.2 (1) 26.7±10.7 (0) 28.8±8.3 (0) 20.8±11.5 (1) 20.9±10.8 (2) 23.8±13.4 (0) 29.7±10.8 (0) 23.5±8.2 (0) 21.8±14.3 (1) 25.8±13.3 (0) 19.2±12.4 (2) 22.8±16.2 (2) 26.6±10.4 (1) 21.6±13.5 (1) 29.8±9.3 (0) 25.8±11.7 (1)
AS-Precision(gold_answer) AS-Precision(wikipage) AS-Recall(gold_answer) AS-Recall(wikipage)
5.3±6.1 (6) 2.7±1.7 (5) 7.4±7.4 (6) 34.2±10.3 (0)
6.6±9.8 (6) 1.0 (6) 8.9±8.5 (5) 34.0±11.6 (0)
16.5±14.0 (1) 9.4±4.6 (1) 20.9±11.6 (0) 33.5±9.8 (0)
32
Figure 5: Pairwise Kendall’s τ agreement between estimator rankings induced by ROC-AUC across models, shown separately for each task. Higher values indicate that the same estimators tend to rank similarly across models for a given hallucination type.
Table 8: Within-panel Spearman rank correlations between evaluation metrics, computed over uncertainty estimators. AlignScore variants are excluded. Task
Model
RagTruth – – PreciseWikiQA – – LongWiki – – NonExistentRefusal – –
Llama2-13b-chat Llama2-7b-chat Mistral-7B-instruct Llama2-13b-chat Llama2-7b-chat Mistral-7B-instruct Llama2-13b-chat Llama2-7b-chat Mistral-7B-instruct Llama2-13b-chat Llama2-7b-chat Mistral-7B-instruct-v0.2
Median Range
– –
33
AUROC–PRR
AUROC–RCE
PRR–RCE
.96 .94 .95 .97 .82 .98 .99 .98 .97 .94 .94 .99
-.99 -.97 -.94 -.96 -.79 -.81 -.98 -.96 -.99 -.94 -.97 -.97
-.95 -.91 -.86 -.95 -.94 -.88 -.97 -.97 -.97 -.88 -.90 -.96
.97 [.82,.99]
-.96 [-.99,-.79]
-.95 [-.97,-.86]
Figure 6: Pairwise Kendall’s τ agreement between estimator rankings induced by ROC-AUC across tasks, shown separately for each model. Higher values indicate that the same estimators tend to rank similarly across hallucination types for a given model.
Table 9: Dataset quality summary across models and tasks. H=Human annotation, J=Judge LLM. Model
Llama 2 7B Chat Llama 2 13B Chat Mistral 7B Instruct
RAGTruth % Hallu. (H) ↓
PreciseWiki % Hallu. (J) ↓
LongWiki F1@32 (J) ↑
NonExistent % Non-refusal (J) ↓
61.6 56.7 66.8
93.5 87.5 86.6
0.38 ± 0.19 0.31 ± 0.16 0.29 ± 0.16
96.4 83.2 93.8
34
LongWiki
True Positive Rate
1.0
NonExistent
0.8 0.6 0.545 0.572 0.506 0.491 0.483 0.585 0.619
0.4 0.2 0.0
0.0
0.2
0.4
0.6
0.8
1.0 0.0
0.525 0.503 0.549 0.375 0.434 0.444 0.658
0.2
False Positive Rate Information-based Sampling-based
0.4
0.6
0.8
1.0
False Positive Rate Internal State Training-based
Reflexive Black-box
AlignScore
Figure 7: Family-level ROC aggregates per task, averaged across the three models. For each task– model panel, estimator ROC curves are first averaged within each family by taking the mean true positive rate at each false positive rate. For each task, these model-level family means are then averaged across models. Shaded bands show one standard deviation of these model-level familymean true positive rates at each false positive rate; they reflect cross-model variability.
35