arXiv:2605.05166v1 [cs.CL] 6 May 2026
T HE F IRST T OKEN K NOWS : S INGLE -D ECODE C ONFIDENCE FOR H ALLUCINATION D ETECTION
Mina Gabriel Department of Computer and Information Sciences Temple University Philadelphia, PA 19122, USA [email protected]
May 7, 2026
A BSTRACT Self-consistency detects hallucinations by generating multiple sampled answers to a question and measuring surface-form agreement, a strategy that often breaks down when answers are semantically similar but lexically different. Semantic self-consistency extends this idea by producing multiple diverse candidate answers per question and using a natural language inference (NLI) model to cluster them by meaning. This method requires repeated sampling and additional inference; a typical setup uses one greedy decode plus ten sampled generations per question, followed by NLI-based aggregation to compute semantic agreement. We show that first-token confidence (ϕfirst )—the normalized entropy of the top-K logits at the first content-bearing answer token of a single greedy decode—matches or modestly exceeds semantic self-consistency on closed-book short-answer factual QA at roughly 1/11 the generation cost, even before accounting for the extra NLI computation overhead. Across three 7–8B instruction-tuned models (Llama-3.1-8B, Mistral-7B-v0.3, Qwen2.5-7B) and two benchmarks (PopQA and TriviaQA, n = 1000 each), ϕfirst achieved a mean AUROC of 0.820, compared with 0.793 for semantic agreement and 0.791 for standard surface-form self-consistency. A subsumption test shows that ϕfirst is moderately to strongly correlated with semantic agreement (Pearson 0.54– 0.76), and a logistic ensemble of the two yields only a +0.02 AUROC improvement over ϕfirst alone, indicating that single-decode confidence captures most of semantic agreement’s discriminative power. Partial-correlation analysis further shows that the apparent association between ϕfirst and answer length largely disappears after controlling for correctness. We argue that first-token confidence should be reported as a default, low-cost baseline before invoking sampling-based uncertainty estimation.
1
Introduction
A common paradigm for uncertainty quantification in large language models is self-consistency: sample N responses for the same input and use disagreement among them as a proxy for uncertainty. Originally proposed as a decoding strategy for reasoning [1], the same sampling-based principle has become central to several hallucination-detection methods. Semantic uncertainty refines this idea by clustering generations into NLI-based equivalence classes and treating disagreement among clusters as evidence of model uncertainty [2, 3]. These methods provide strong baselines, but require multiple generations per question and a separate NLI-based clustering model. We argue that, in closed-book short-answer factual QA, where the model answers from its parametric knowledge without retrieved documents, sampling-based methods act as expensive Monte Carlo probes of uncertainty that is already largely visible in the model’s first-token logit distribution. For factual questions such as “Who wrote Hamlet?” or “What is the capital of Australia?”, the first generated answer token often marks the model’s earliest commitment to an entity, name, or relation value. If most of the probability mass at this position is concentrated on one token, the model is making a confident early choice about how to begin the answer. If the probability mass is instead spread across several plausible first tokens, the model is unsure which answer to begin generating, even before the rest of the response has unfolded.
A PREPRINT - M AY 7, 2026
We define first-token confidence ϕfirst as the normalized entropy of the top-K logits at the first content-bearing answer token of a single greedy decode and compare it against semantic self-consistency, surface-form self-consistency, and verbalized confidence. We further test whether ϕfirst captures much of the same uncertainty information as semantic agreement, which requires multiple sampled generations. Our contributions are: (i) we show that ϕfirst matches or modestly exceeds semantic agreement on PopQA and TriviaQA across three 7–8B models, at roughly 1/11 of the generation cost, before accounting for the additional NLI clustering required by semantic agreement; (ii) we provide a subsumption test showing that ϕfirst is moderately to strongly correlated with semantic agreement and that a logistic ensemble of the two adds only marginal AUROC over ϕfirst alone; and (iii) we show that the apparent relationship between ϕfirst and answer length is largely explained by correctness rather than answer length itself.
2
Method
2.1
First-token confidence
Given a single greedy decode of a model’s response, let ℓt ∈ R|V | denote the logits at decode step t and pt,i the corresponding softmax probabilities. Let t⋆ be the position of the first content-bearing answer token, identified by skipping whitespace, punctuation, and chat-template prefixes such as “Answer:”. We take the top-K probabilities at position t⋆ (with K = 100), renormalize them to p̃t⋆ ,1 , . . . , p̃t⋆ ,K , and define Ht⋆ = −
K X
p̃t⋆ ,i log p̃t⋆ ,i ,
i=1
ϕfirst = 1 −
Ht⋆ . log K
ϕfirst ranges from 0 (uniform top-K) to 1 (all mass on a single token). It is computed from a single greedy forward pass: no additional sampling, no external models. 2.2
Uncertainty baselines
We sample N = 10 completions per question using temperature 0.7 and top-p = 0.95. AU-full measures surface-form agreement by computing the fraction of sampled completions whose normalized full strings match the normalized greedy answer. AU-3w and AU-1w progressively relax this criterion to the first three words and the first word, providing increasingly strong surface-form baselines. Semantic AU performs meaning-level agreement by clustering the greedy answer and its N samples using bidirectional NLI entailment with DeBERTa-v3-large-mnli [4], following the procedure of [2], and reports the fraction of samples assigned to the greedy answer’s cluster. Verbalized confidence prompts the model to output an integer from 0–100 reflecting its self-estimated correctness [5, 6]. We use the same sampling hyperparameters and scoring rules across all datasets and models. The resulting AUROC values should therefore be interpreted as untuned estimates rather than benchmark-specific optimized results. 2.3
Cost
ϕfirst requires one greedy forward pass per question. Semantic AU requires one greedy decode, N = 10 sampled generations, and representative-based bidirectional NLI clustering over the greedy and sampled answers. This requires O(CN ) NLI comparisons, where C is the number of discovered semantic clusters.
3
Experiments
3.1
Setup
We evaluate on the test split of PopQA [7] and the validation split of TriviaQA [8], sampling n = 1000 examples per dataset with a fixed seed. The same 1000 examples are used across all three models so that all comparisons are paired at the example level. √ We choose n = 1000 as a compute–precision tradeoff. The standard error of an AUROC estimate decreases as Θ(1/ n), so doubling to n = 2000 would only narrow each cell’s bootstrap interval by about 0.007 AUROC points, while doubling all generation and NLI costs. We instead invest the saved compute in three models, two datasets, and the multi-method comparison, and report empirical 95% bootstrap confidence intervals and paired bootstrap tests for every cell. We evaluate three instruction-tuned 7–8B models: Llama-3.1-8B-Instruct [9], Mistral-7B-Instruct-v0.3 [10], and Qwen2.5-7B-Instruct [11]. Correctness is determined by an automatic judge (Qwen2.5-14B-Instruct in 4-bit) given the question, the model’s answer, and gold aliases. 2
A PREPRINT - M AY 7, 2026
(a) AUROC by (dataset, model) cell
(b) AUROC grid
0.9
0.70
0.78
0.61
0.70
0.77
AU-1w
0.84
0.73
0.83
0.75
0.63
0.72
AU-3w
0.85
0.76
0.86
0.77
0.70
0.75
AU-full
0.87
0.78
0.86
0.77
0.72
0.76
Sem-AU
0.87
0.77
0.87
0.78
0.72
0.74
first
0.89
0.84
0.90
0.79
0.73
0.77
Llama PopQA
Mistral PopQA
Qwen Llama Mistral Qwen PopQA TriviaQA TriviaQA TriviaQA
0.90 0.85 0.80
0.7
0.6
0.75
AUROC
AUROC
0.8
Verbalized
0.63
0.70 0.65 0.60
0.5 Llama PopQA
Mistral PopQA
Verbalized
Qwen PopQA AU-1w
AU-3w
Llama TriviaQA AU-full
Mistral TriviaQA
Qwen TriviaQA
Sem-AU
0.55
first
Figure 1: AUROC of correctness prediction across six dataset–model cells and six confidence signals (n=1000 per cell). (a) Grouped bars show AUROC per cell; the dashed line marks chance performance. (b) The same values are shown as a heatmap. ϕfirst achieves the highest AUROC in five of six cells and is within 0.002 AUROC of the strongest method in the remaining cell, while requiring only a single greedy decode. 3.2
Main results
In this subsection, we compare ϕfirst with verbalized confidence, surface-form self-consistency, and semantic selfconsistency. The main question is whether a single-decode token-level confidence signal can match or exceed uncertainty signals that require multiple sampled generations. Table 1: AUROC for hallucination detection across three 7–8B instruction-tuned models on PopQA and TriviaQA (n = 1000 each). Methods are grouped by inference cost. Best per row in bold; second-best underlined. ∆ is the gap between ϕfirst and the strongest non-ϕ baseline. 1+10 decodes (sampling)
1 decode
1 decode
Dataset
Model
Verb.
AU-1w
AU-3w
AU-full
Sem. AU
ϕfirst
∆
PopQA
Llama-3.1-8B Mistral-7B-v0.3 Qwen2.5-7B
0.632 0.701 0.782
0.840 0.735 0.831
0.854 0.762 0.856
0.866 0.778 0.857
0.874 0.775 0.867
0.887 0.842 0.895
+0.013 0.27 +0.064 0.25 +0.028 0.19
TriviaQA
Llama-3.1-8B Mistral-7B-v0.3 Qwen2.5-7B
0.614 0.696 0.774
0.752 0.632 0.725
0.770 0.701 0.751
0.769 0.718 0.758
0.778 0.724 0.741
0.794 0.727 0.772
+0.016 0.64 +0.003 0.62 −0.002 0.52
PopQA mean TriviaQA mean
0.705 0.695
0.802 0.703
0.824 0.741
0.834 0.748
0.839 0.748
0.875 0.764
+0.036 0.24 +0.016 0.59
Overall mean
0.700
0.752
0.782
0.791
0.793
0.820
+0.027 0.42
Acc.
Figure 1 summarizes our main result visually, and Table 1 reports the corresponding numbers. Panel (a) of Figure 1 shows AUROC as grouped bars per dataset–model cell, with ϕfirst highlighted; panel (b) presents the same values as a heatmap for at-a-glance comparison across methods. Both views show the same pattern: ϕfirst is the strongest method in five of six dataset–model cells and is within 0.002 AUROC of the strongest method in the remaining cell. The pattern is consistent across both datasets: ϕfirst improves the per-dataset mean by +0.036 AUROC on PopQA (0.875 vs. 0.839 for semantic AU) and by +0.016 on TriviaQA (0.764 vs. 0.748). The smaller TriviaQA gain suggests that longer and more lexically variable answers give sampling-based methods relatively more opportunity to recover useful agreement information; we return to this point in the limitations. In the overall mean, ϕfirst reaches 0.820 AUROC, compared with 0.793 for semantic AU, 0.791 for AU-full, 0.782 for AU-3w, and 0.752 for AU-1w. Verbalized confidence is weaker, with a mean AUROC of 0.700, consistent with prior work showing that LLMs are often poorly calibrated when asked to state their own confidence directly [5]. Thus, the 3
A PREPRINT - M AY 7, 2026
advantage of ϕfirst over semantic AU is modest in absolute terms (+2.7 AUROC points on average), but it is obtained with a single greedy decode rather than multiple sampled generations and NLI-based semantic clustering. 3.3
Statistical reliability of the gains
The AUROC results show the size of the performance differences, but do not show by themselves whether those differences are stable between evaluation examples. We therefore use paired bootstrap resampling over questions to compare ϕfirst against the main baselines within each dataset–model cell. Because both methods are evaluated on the same questions, the test measures whether the observed AUROC gap is robust to resampling of the evaluation set. Table 2: Paired bootstrap test of ∆AUROC > 0 for ϕfirst vs. each baseline (B=1000 resamples; one-sided p). With B = 1000, the smallest resolvable p-value is ≈ 0.001, so cells reported as < 0.001 correspond to 0/1000 resamples favoring the baseline. Bold indicates p < 0.05. Dataset
Model
vs. AU-full
vs. Sem. AU
vs. AU-1w
PopQA
Llama Mistral Qwen
0.018 <0.001 0.001
0.082 <0.001 0.008
<0.001 <0.001 <0.001
TriviaQA
Llama Mistral Qwen
0.025 0.301 0.153
0.105 0.416 0.014
<0.001 <0.001 <0.001
4/6
3/6
6/6
Significant (p < 0.05)
Table 2 reports paired bootstrap tests over questions. These tests ask whether the AUROC advantage of ϕfirst over each baseline is stable under resampling of the same evaluation examples. The results show that ϕfirst significantly outperforms AU-full in four of six cells and semantic AU in three of six cells. The remaining semantic-AU differences are not statistically significant, so we frame ϕfirst as matching semantic self-consistency rather than uniformly outperforming it. Against AU-1w, the simplest surface-form baseline, the gain is significant in all six cells. 3.4
Subsumption analysis
We test whether ϕfirst already captures the information provided by semantic AU. For each cell we report two quantities: the Pearson correlation between ϕfirst and semantic AU, and the AUROC gain obtained by combining both signals in a standardized logistic regression over ϕfirst alone. A high correlation paired with a near-zero ensemble gain indicates that semantic AU adds little beyond ϕfirst . Table 3: Subsumption analysis. Pearson r: correlation between ϕfirst and semantic AU. Gain: AUROC of the logistic ensemble of both signals minus AUROC of ϕfirst alone. Small gains indicate semantic AU adds little beyond ϕfirst . Dataset
Model
Pearson r
Ensemble gain
PopQA
Llama Mistral Qwen
0.76 0.59 0.75
+0.017 +0.009 +0.012
TriviaQA
Llama Mistral Qwen
0.74 0.54 0.67
+0.019 +0.045 +0.024
0.67
+0.021
Mean
Three observations follow from Table 3. First, ϕfirst and semantic AU are moderately to strongly correlated, with Pearson r between 0.54 and 0.76 (mean 0.67). Second, combining the two signals improves AUROC by only +0.021 on average, and by less than +0.025 in five of six cells. Third, ϕfirst alone matches or exceeds semantic AU’s standalone AUROC in every cell, so the residual ensemble gain reflects a small complementary contribution from semantic AU rather than a deficit in ϕfirst . Together, these results indicate that ϕfirst already captures most of the discriminative content that semantic agreement extracts at substantially higher inference cost. 4
A PREPRINT - M AY 7, 2026
3.5
Length confound
A natural concern is that ϕfirst may simply track the length of the generated answer. We test this in two stages. First, we compute the raw Pearson correlation rlen between ϕfirst and the number of generated answer tokens. Second, since wrong answers tend to be both longer and lower-confidence, we control for correctness by computing the partial Pearson partial correlation rlen between ϕfirst and answer length after removing the linear effect of the binary correctness label from both variables. partial Table 4: Length confound. rlen : raw Pearson correlation between ϕfirst and answer length. rlen : partial correlation controlling for correctness. Values close to zero after partialling indicate that length is not driving ϕfirst .
Dataset
Model
rlen
partial rlen
PopQA
Llama Mistral Qwen
−0.16 −0.13 −0.14
−0.02 −0.03 −0.04
TriviaQA
Llama Mistral Qwen
−0.23 −0.25 −0.11
−0.18 −0.17 −0.05
Table 4 reports both quantities. The raw correlation ranges from −0.11 to −0.25 across cells, accounting for at most 6.5% of the variance in ϕfirst . On PopQA, the partial correlation shrinks substantially: from −0.16 to −0.02 for Llama and from −0.13 to −0.03 for Mistral. This suggests that the apparent length effect on PopQA is largely explained by correctness rather than answer length itself. On TriviaQA, the partial correlation drops by less: a residual correlation of about −0.18 remains for Llama and Mistral. This indicates a small but non-trivial residual sensitivity to answer length on TriviaQA, which we list as a limitation.
4
Related work
Semantic self-consistency [2, 3] estimates uncertainty from disagreement among NLI-based equivalence classes of multiple sampled generations. Surface-form variants compute agreement of normalized strings or first words. Singlepass alternatives include token-level probabilities, sequence-level likelihood [12], model-internal probes [13, 14], and verbalized confidence [5, 6]. To our knowledge, no prior work directly evaluates first-token entropy as a standalone hallucination signal against semantic self-consistency, nor quantifies how much of the semantic-agreement signal is already encoded in single-decode confidence.
5
Discussion and conclusion
First-token confidence matches or modestly exceeds semantic self-consistency on closed-book factual QA across three 7–8B instruction-tuned models, at roughly 1/11 of the generation cost, before accounting for the additional NLI clustering required by semantic agreement. The subsumption test shows that ϕfirst is moderately to strongly correlated with semantic agreement and recovers most of its discriminative content from a single greedy decode. We recommend that future hallucination-detection methods report ϕfirst as a default cheap baseline before claiming gains from sampling-based methods. Limitations. Our study is restricted to English closed-book short-answer factual QA with three open 7–8B models and two benchmarks at n = 1000 each. The results may not transfer to long-form generation, multi-hop or reasoning-heavy QA, retrieval-augmented settings, multilingual QA, larger or proprietary models, or black-box APIs that do not expose token probabilities. The method requires logits at the first answer-token position; reliable identification of that position depends on the chat template and tokenizer. We observed in preliminary analysis that aggregating confidence across all generated tokens can recover additional signal on TriviaQA, suggesting that ϕfirst does not exhaust what single-decode probabilities offer; we leave fuller aggregation methods to future work. Some residual length sensitivity remains on TriviaQA after controlling for correctness, suggesting that length-related artifacts cannot be ruled out entirely. Finally, our correctness labels come from an automatic judge rather than human annotation, so a small amount of label noise may propagate into the reported AUROCs. 5
A PREPRINT - M AY 7, 2026
References [1] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations (ICLR), 2023. [2] Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In International Conference on Learning Representations (ICLR), 2023. [3] Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Generating with confidence: Uncertainty quantification for black-box large language models. Transactions on Machine Learning Research, 2024. [4] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. DeBERTa: Decoding-enhanced BERT with disentangled attention. In International Conference on Learning Representations (ICLR), 2021. [5] Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D. Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023. [6] Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs. arXiv preprint arXiv:2306.13063, 2023. [7] Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), 2023. [8] Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), 2017. [9] Abhimanyu Dubey et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. [10] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7B. arXiv preprint arXiv:2310.06825, 2023. [11] An Yang et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024. [12] Andrey Malinin and Mark Gales. Uncertainty estimation in autoregressive structured prediction. In International Conference on Learning Representations (ICLR), 2021. [13] Saurav Kadavath, Tom Conerly, Amanda Askell, et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022. [14] Amos Azaria and Tom Mitchell. The internal state of an LLM knows when it’s lying. In Findings of the Association for Computational Linguistics: EMNLP, 2023.
6