ORCE: Order-Aware Alignment of Verbalized Confidence in Large Language Models
arXiv:2605.12446v1 [cs.LG] 12 May 2026
Chen Li1
Xiaoling Hu2 Songzhu Zheng3 Jiawei Zhou1 Chao Chen1 1 Stony Brook University, NY, USA 2 Massachusetts General Hospital and Harvard Medical School, MA, USA 3 Morgan Stanley, NY, USA
Abstract Large language models (LLMs) often produce answers with high certainty even when they are incorrect, making reliable confidence estimation essential for deployment in real-world scenarios. Verbalized confidence, where models explicitly state their confidence in natural language, provides a flexible and user-facing uncertainty signal that can be applied even when token logits are unavailable. However, existing verbalized-confidence methods often optimize answer generation and confidence generation jointly, which can cause confidence-alignment objectives to interfere with answer accuracy. In this work, we propose a decoupled and order-aware framework for verbalized confidence calibration. Our method first generates an answer and then estimates confidence conditioned on the fixed question–answer pair, allowing confidence optimization without directly perturbing the answer-generation process. To align confidence with correctness likelihood, we construct a sampling-based surrogate from multiple model completions and optimize rank-based reinforcement learning objectives that encourage responses with higher estimated correctness likelihood to receive higher verbalized confidence. Experiments on reasoning and knowledge-intensive benchmarks show that our method improves calibration and failure prediction performance while largely preserving answer accuracy. These results demonstrate that verbalized confidence can be more reliably aligned by decoupling confidence estimation from answer generation and optimizing the relative ordering of confidence across responses.
1
Introduction
Large language models (LLMs) are increasingly deployed in high-stakes domains where the quality and reliability of generated answers are critical, such as law [17] and healthcare [18]. In these settings, correctness alone is not enough: models must also know when and how to express uncertainty. Yet current LLMs are not explicitly optimized to produce reliable confidence estimates, and their verbalized confidence often remains high even when their responses are incorrect [10, 26, 9, 33]. Such overconfidence undermines trust building and remains a key obstacle for the deployment of LLMs in safety-critical applications. A natural approach to confidence estimation is to use the model’s internal probabilities or logits assigned to generated answers [1, 13, 14]. Because LLMs generate text autoregressively, such probabilities are defined over tokens conditioned on previous tokens, making logit-based confidence a form of tokenized confidence. However, tokenized confidence measures the likelihood of a specific token sequence rather than the semantic correctness of the response. This is poorly suited to free-form generation, where semantically equivalent answers may have different wordings and therefore different token likelihoods. They are also unavailable for many black-box LLMs. Verbalized confidence, in which the model explicitly states its confidence in natural language, Email: Chen Li ([email protected]).
Preprint.
provides a flexible and user-facing alternative that applies across model types and task formats. Yet existing studies show that verbalized confidence is frequently miscalibrated, overconfident, and highly sensitive to prompting. Prompt-based strategies [30, 33] can partially reduce overconfidence, but their improvements are limited, motivating explicit optimization of stated confidence. Separately, ensemble-style methods estimate uncertainty from group-level signals such as consistency across multiple sampled responses [33]; however, they require repeated generation at inference time and can introduce additional variance from stochastic sampling. These limitations motivate a method that preserves the user-facing benefits of verbalized confidence while aligning it more directly with response correctness. Recent methods have attempted to improve verbalized confidence through supervised fine-tuning (SFT) [19], or reinforcement learning (RL) [34, 28]. However, existing approaches jointly optimize answer generation and confidence generation. This coupling creates a crucial conflict: objectives designed to improve uncertainty estimation may alter the model’s answer distribution and degrade task accuracy (Appendix F.6 of [19]). In other words, improving confidence calibration should not come at the cost of generating incorrect answers. This issue is particularly problematic for user-facing LLM systems, where confidence estimates are only useful if they accompany strong answers. In this work, we propose a simple but effective decoupled framework for verbalized confidence estimation Instead of training the model to generate answers and confidence jointly, we decouple the process into two stages: the model first produces an answer, and then generates a confidence estimate conditioned on the fixed question–answer pair. This design treats the generated answer as an input for confidence estimation, allowing us to optimize confidence behavior without directly perturbing the answer-generation process. As a result, our method improves the reliability of verbalized confidence while better preserving the model’s original task accuracy. The method is built on two key ideas. First, we estimate the uncertainty of the answer model through a sampling-based correctness likelihood. For each prompt, we sample multiple completions from the answer model and use their empirical correctness frequency as a surrogate for how likely the model is to answer that prompt correctly. This provides a model-agnostic signal of answer reliability without requiring access to internal logits or token probabilities. Second, we use this surrogate to train verbalized confidence with an order-aware objective. Rather than only matching absolute confidence values, we optimize rank-based objectives that encourage the confidence model to preserve the relative ordering induced by the surrogate: responses associated with higher estimated correctness likelihood should receive higher verbalized confidence. This formulation is well suited to confidence estimation, where relative reliability is often more robust and practically more meaningful than exact numerical calibration.We evaluate our method on a diverse set of knowledge-intensive reasoning, discrete reasoning, and logical reasoning tasks. Across these settings, our method yields promising improvements on both calibration and failure-prediction metrics. Our contributions are summarized as follows. 1. We identify the coupling between answer generation and confidence generation as an important limitation of existing verbalized-confidence alignment methods, and propose a two-stage framework that decouples the two. 2. We introduce a sampling-based surrogate of correctness likelihood for supervising verbalized confidence without relying on internal model probabilities. 3. We propose an order-aware reinforcement learning objective that aligns verbalized confidence with this surrogate through rank-based optimization. 4. We show that our method improves confidence calibration and failure prediction performance across reasoning and knowledge-intensive benchmarks while largely preserving answer accuracy. Together, these results suggest that verbalized confidence can be made more reliable by optimizing not only what confidence value the model states, but also how confidence is ordered across responses of different reliability. 2
Prompt confidence estimation
Answer generation
Prompt x
Prompt x
𝐴1 𝜅𝑠 𝑥 =
𝐴2 Frozen answer LLM
Sample K answers from answer LLM
Alignment via DPO
1 𝐼{𝑌𝑖 = 𝑦 ∗ } 𝐾 𝑖
Monte Carlo estimate of prompt-level confidence
… 𝐴𝐾
Verbalized confidence (answer) generation Generated answer y
Parameters fixed during alignment
Confidence LLM (generating candidates)
Prompt x
Confidence LLM
Generated answer y
Verbalized confidence 𝐶(𝑥, 𝑦) “I am 72% confidence”
𝑧1 55%
𝑧2 72%
𝑧3 88%
𝑅𝑆𝐶 𝑐, 𝜅; 𝐶, 𝐾 = 𝑟𝑠 𝐶 ∪ 𝑐 , 𝐾 ∪ 𝜅
Preferred 𝑧+ (e.g., 88%)
…
− 𝑟𝑠 (𝑋, 𝐾)
Rejected 𝑧+ (e.g., 55%)
DPO preference optimization (minimize DPO loss)
Figure 1: An overview of our method: ORCE separates answer generation from confidence alignment, uses an ensemble-based surrogate κs (x), and optimizes the confidence LLM with a Spearmancorrelation reward.
2
Related Work
Confidence estimation in classical deep learning. Confidence estimation is widely studied [3, 15, 7, 25, 24, 23, 5, 22, 12, 38] in classical deep learning, such as image classification and segmentation tasks. Surrogate based methods [23, 16] use the correctness frequency or consistency as a surrogate for confidence estimation. Dropout [3] approximates predictive uncertainty by performing multiple stochastic forward passes with dropout enabled and measuring the mean prediction and variability across the resulting outputs. Deep ensembles [15] estimate confidence by training multiple independently initialized neural networks and aggregating their predictions to obtain both the final prediction and its uncertainty. Because neural networks produce logit or probability outputs, classical confidence estimation methods can conveniently derive confidence scores directly from the model’s predictive distribution. Confidence estimation in LLMs. Prior work estimates LLM confidence from answer logits or probabilities [1, 13, 14]. While these internal scores are useful, they are tied to the likelihood of specific token sequences and are difficult to define for free-form responses with many semantically equivalent surface forms. Verbalized confidence offers a more flexible and user-facing alternative: it can be elicited without logit access, applies to black-box models, and more directly communicates uncertainty over the semantic correctness of generated answers. Another line of work estimates confidence through self-consistency or ensembles by sampling multiple completions and aggregating agreement across responses [33, 31]. These methods are model-agnostic and capture uncertainty over the answer distribution, but require multiple generations at inference time and do not directly produce a single user-facing confidence statement. Recent studies investigate verbalized confidence, where LLMs explicitly state their confidence in natural language [20, 30, 33, 28]. Prompting-based studies, which ask language models to explicitly report how confident they are in their answers, show that verbalized confidence can be informative, but is often overconfident, prompt-sensitive, and imperfectly aligned with model uncertainty [33, 36]. Beyond prompting, LACIE [28] calibrates confidence through listener-aware preference optimization, SaySelf [34] trains models to generate confidence with self-reflective rationales, ConfTuner [19] optimizes a tokenized Brier-score objective, and CONQORD [29] aligns verbalized confidence with response quality using reinforcement learning. In contrast to these methods, which typically optimize answer and confidence generation jointly, our approach decouples the two stages: the model first generates an answer and then estimates confidence conditioned on the fixed response. This design enables order-aware alignment between verbalized confidence and estimated correctness likelihood while reducing interference with answer-generation accuracy. 3
3
Method
We introduce ORCE, an order-aware reinforcement learning method for verbalized confidence estimation in language models. Our method decouples the answer and confidence generation into two stages. The overview of our method is shown in Figure 1. Decoupling answer and confidence generation. In prior studies [33, 29], verbalized confidence is generated jointly with the answer using the same model: given a prompt, the LLM outputs both the answer and its associated confidence statement. In Miao et al. [33], this generation strategy is applied only at inference, making it simple and computationally efficient. However, when reinforcement learning (RL) fine-tuning is introduced [29], joint generation conflates two distinct objectives. When the policy seeks to improve outputs, the model cannot distinguish whether the improvement signal pertains to the answer or to the verbalized confidence, so RL updates intended to recalibrate confidence can inadvertently degrade answer accuracy. This issue is exacerbated by the limited size of confidencealignment datasets relative to the pretraining corpus. To address this, we decouple generation and confidence calibration into two separate stages. In the first stage, an answer LLM produces only the answer, so its parameters are never touched by confidence-alignment updates and answer accuracy is preserved by construction. In the second stage, a separate confidence LLM generates a verbalized confidence conditioned on both the prompt and the previously generated answer. The confidence LLM is optimized via reinforcement learning. From prompt-level confidence to answer-level confidence. Once the Answer LLM is optimized, the Confidence LLM estimates the verbalized confidence C(x, y) for each prompt-answer pair (x, y). A primary challenge in training a reliable estimator is the absence of ground-truth labels; the model’s internal certainty for a specific pair is not directly observable. However, we can approximate this at the prompt level by generating multiple answers for a single prompt and measuring their correctness against a ground-truth answer; such surrogate has been used in confidence estimation literature [23]. We leverage these prompt-level confidences as coarse supervision for the Confidence LLM, which uses RL to align its verbalized confidence (at answer level). During training, the Confidence LLM generates confidence scores for a curated list of prompt-answer pairs. This list and the prompt-level ground truth confidences are both sorted and compared using the Spearman correlation (measuring their ranking similarity). To optimize this alignment, we employ DPO. Specifically, we iteratively generate new candidate answers and accept or reject the resulting pairs based on their ability to improve the ranking similarity, ultimately yielding a robust prompt-answer confidence estimator. 3.1
Prompt Level Confidence Estimation
For each prompt x, let y ⋆ denote the reference answer, and let g(y, y ⋆ ) ∈ {0, 1} be a task-specific correctness function indicating whether an answer y should be regarded as correct. For multiplechoice tasks, g(y, y ⋆ ) = I{y = y ⋆ } reduces to exact match. For free-form tasks, g is instantiated using the task’s standard evaluation criterion, such as normalized exact match, answer extraction, or other benchmark-specific equivalence rules. Using this correctness function, we can define the prompt-level reliability of the answer LLM as, η(x) := P g(Ŷ , y ⋆ ) = 1 | x , where Ŷ is the answer produced by the answer LLM. η(x) measures the marginal probability that the answer model produces a correct answer on prompt x. Since η(x) is not directly accessible, we introduce an ensemble-based surrogate. For each prompt, we sample K candidate answers {Yi }K i=1 from the answer LLM at non-zero temperature and define, K
κs (x) =
1 X g(Yi , y ⋆ ). K i=1
By construction, κs (x) is an unbiased Monte Carlo estimator of η(x): E[κs (x) | x] = η(x), with variance η(x)(1 − η(x))/K. Therefore, κs (x) provides a consistent finite-sample estimate of the prompt-level reliability of the answer model. Selection of prompt-answer pairs. We generate prompt-answer pairs for the confidence LLM to estimate confidences, and to align them via DPO. The choice of these pairs, however, cannot be arbitrary. After estimating κs (x) from the K sampled answers, we construct the realized answer 4
Shared setup
Local reward: NRD
New prompt Surrogate 𝜿 = 𝟎. 𝟑𝟓 Confidence Candidate a: c=0.25
Candidate b: c=0.45 Confidence
Surrogate
1
0.2
0.1
2
0.3
0.3
3
0.4
0.5
4
0.5
0.4
5
0.6
0.6
Candidate a confidence: 0.25
Candidate b confidence: 0.45
0.2
0.4
0.3
0.5
Global reward: SC 𝑅𝑆𝐶 𝑐, 𝜅, 𝐶, 𝐾 = 𝑟𝑠 𝐶 ∪ 𝑐 , 𝐾 ∪ 𝜅
− rs (C, K)
0.6
Rank
1
2
3
4
5
Surrogate
0.1
0.3
0.4
0.5
0.6 Candidate a: c=0.25 𝑅𝑆𝐶 = −0.0143 Higher reward
New prompt surrogate 0.35
𝑅𝑁𝑅𝐷 𝑐𝑖 , 𝜅𝑖 = −|𝑟𝑎𝑛𝑘 𝑐𝑖 − 𝑟𝑎𝑛𝑘(𝜅𝑖 )|
NRD treats two candidate the same .
Candidate a: c=0.45 𝑅𝑆𝐶 = −0.0714 Lower reward
Spearman correlation reward prefers candidates with higher global rank correlation
Figure 2: Comparison between local rank matching (NRD) and global Spearman alignment. Although the two candidate confidences have similar local rank discrepancy under NRD, the SC reward prefers c(a) = 0.25 because it better preserves the global monotone relationship between verbalized confidence and surrogate reliability.
according to the empirical majority-correctness regime. Concretely, if κs (x) ≥ 1/2, we pair the prompt with a correct sampled answer; if κs (x) < 1/2, we pair it with an incorrect sampled answer. In this way, the realized answer presented to the confidence model reflects the empirical correctness tendency of the answer model on that prompt, allowing the answer-conditioned score C(x, y) to be trained against a prompt-level surrogate target. 3.2
Order-aware reward via Spearman correlation
Order-aware principle. We adopt an order-based criterion for verbalized confidence at the level of the constructed training distribution. For each prompt x, we first estimate its prompt-level reliability using the ensemble surrogate κs (x), and then construct the realized answer y so that its correctness state matches the majority-correctness regime implied by κs (x). As a result, prompts with large surrogate reliability are paired with correct realized answers, whereas prompts with small surrogate reliability are paired with incorrect realized answers. Under this construction, we train the confidence model so that, across training instances, C(xi , yi ) < C(xj , yj ) ⇐⇒ κs (xi ) < κs (xj ), up to finite-sample noise in the surrogate. Thus, although the confidence model is conditioned on realized answers, the ordering signal it is trained to preserve is induced by the prompt-level reliability of the answer model. Given n pairs of verbalized and surrogate confidences {(ci , κi )}ni=1 , let C = {ci }ni=1 , K = {κi }ni=1 , and let rank(ci ) and rank(κi ) denote the ranks of ci and κi within C and K, respectively. A natural question is how to design a reward that encourages rank preservation. One simple local alternative is to compare the rank of each generated confidence value with the rank of its surrogate score, RNRD (ci , κi ) = − |rank(ci ) − rank(κi )| . We refer to this baseline as Numeric Ranking Difference (NRD). Although intuitive, NRD is local: it scores each candidate only by its own rank discrepancy and does not directly evaluate how the candidate affects the overall monotone association between verbalized confidence and surrogate reliability. As a result, two candidates with similar individual rank discrepancies can have different effects on the global confidence–surrogate ordering. To directly optimize the desired order-aware property, we instead use a global rank-correlation reward. Rather than matching each rank independently, a global reward scores a candidate by how much it improves or degrades the overall rank agreement between the verbalized confidence set and the surrogate set. Since our goal is to preserve the full reliability ordering induced by κs , we adopt a Spearman-correlation reward as our primary design. Spearman correlation reward. Spearman’s correlation measures the monotone association between two variables and therefore naturally matches our order-aware objective. The Spearman correlation between C and K is rs (C, K) =
cov[rank(C), rank(K)] . σrank(C) σrank(K) 5
For a new candidate pair (c, κ), we define the Spearman-correlation (SC) reward as its marginal contribution to the global rank correlation, RSC (c, κ; C, K) = rs (C ∪ {c}, K ∪ {κ}) − rs (C, K). Thus, a candidate receives positive reward when adding it improves the monotone agreement between verbalized confidence and surrogate reliability, and negative reward when it disrupts this agreement. A comparison between SC and NRD reward is shown in Figure 2. Warm start via SFT. A limitation of any correlation-based reward is that it can reinforce an incorrect monotone trend if the initial relationship between C and K is negatively correlated or degenerate. To prevent this, we perform a supervised fine-tuning (SFT) stage prior to reinforcement learning, which establishes a positive correlation between verbalized and surrogate confidences. This SFT pretrained model will produce C for the reinforcement learning stage. As we formalize in Section 3.4, this positivity is exactly the condition that ensures the SC reward points in the right direction during RL. 3.3
Alignment strategy
To align the confidence LLM with the surrogate using the SC reward, we adopt Direct Preference Optimization (DPO). DPO is well-matched to our setting: the SC reward naturally produces, for each prompt, a ranking over sampled verbalized confidences, from which preferred/rejected pairs can be extracted without on-policy rollouts or a separate value model. The full alignment procedure is as follows. For each prompt in the training set, we (i) generate an answer using the answer LLM; (ii) sample multiple verbalized confidences from the confidence LLM for the resulting (x, y) pair; (iii) score each sampled confidence c using RSC (c, κs (x); C, K), where (C, K) is the reference set of confidence–surrogate pairs maintained during training; and (iv) designate the highest- and lowest-scoring samples as the preferred (z + ) and rejected (z − ) responses, respectively. Given the resulting preference data (x, z + , z − ), DPO encourages the model to assign higher likelihood to z + than to z − while remaining close to a reference model πref (the SFT confidence LLM prior to RL): LDPO = −E log sig β log πθ (z + | x) − log πθ (z − | x) − log πref (z + | x) + log πref (z − | x) , where πθ is the confidence LLM being optimized, sig(·) is the sigmoid function, and β controls the strength of preference alignment. 3.4
Theoretical analysis of the SC reward
We give a brief interpretation of the SC reward from three perspectives: its ideal population optimum, its surrogate approximation, and the role of the SFT warm-start. Let X be a random prompt, let Y ⋆ be its reference answer, and let Ŷ ∼ πA (· | X) be the answer produced by the answer LLM. Define the correctness indicator Z := g(Ŷ , Y ⋆ ) ∈ {0, 1} and the prompt-level reliability, η(X) := P(Z = 1 | X). The confidence model takes the realized prompt– answer pair (X, Ŷ ) as input and outputs a scalar score S(X, Ŷ ) := fθ (X, Ŷ ) ∈ R. Theorem 1 (Spearman optimum). Assume η(X) has a continuous distribution on (0, 1) and S(X, Ŷ ) has a continuous distribution on R. Then ρS S(X, Ŷ ), η(X) ≤ 1, with equality if and only if there exists a strictly increasing function T : (0, 1) → R such that S(X, Ŷ ) = T η(X) almost surely. Theorem 1 shows that, at the population level, optimizing Spearman alignment recovers a score that is a monotone transform of the prompt-level reliability. Although the confidence model is conditioned on the realized answer, the population-optimal ordering is induced by η(X). (K)
In practice, η(X) is unavailable, so we optimize against the ensemble surrogate κs (X) = PK 1 ⋆ i=1 g(Yi , Y ), where Y1 , . . . , YK are i.i.d. samples from πA (· | X). By the strong law of K (K) large numbers, κs (X) → η(X) almost surely as K → ∞. Under mild continuity assumptions, this implies ρS S(X, Ŷ ), κ(K) s (X) → ρS S(X, Ŷ ), η(X) , so SC alignment against the surrogate approaches SC alignment against the ideal prompt-level target. 6
Finally, the SC reward is defined relative to a reference set (C, K) through ∆SC (c, κ; C, K) = rs (C ∪ {c}, K ∪ {κ}) − rs (C, K). Hence it acts as an update signal for improving a global rank objective around the current anchor set. Because the SFT-initialized reference set is generally imperfect, warm-start is important: when the initial alignment is positive and the misordering is moderate, SC tends to favor candidates that better preserve the surrogate-induced ordering; if the anchor is severely misaligned, the same relative mechanism can reinforce the wrong trend. Thus, the role of SFT is not to produce a perfect ranking, but to initialize the model in a sufficiently aligned regime where global rank refinement is reliable. Proofs and additional discussion are deferred to Appendix A.
4
Experiment
In this section, we evaluate the confidence estimation performance of our method on three widely used open-weight foundation models: Llama-3-8B [6], Qwen3-8B [35], and Mistral-7B-v0.3 [11]. These models span different model families and training recipes, allowing us to assess whether the proposed decoupled and order-aware confidence alignment framework generalizes across diverse LLM backbones. Benchmarks. We evaluate our method on three representative reasoning settings: MMLU [8] for knowledge-intensive reasoning, DROP [2] for discrete reasoning, and LogiQA 2.0 [21] for logical reasoning. For logical reasoning, we train on LogiQA 2.0 and additionally report out-of-domain generalization results on ReClor [37]. Results on the LogiQA 2.0 test set are provided in Appendix B, and further experimental details are given in Appendix C. Baselines. We compare against representative prompting- and training-based baselines. Vanilla uses our decoupled answer-conditioned prompting format without any training. Self-Consistency [33] estimates confidence from agreement across multiple sampled responses. Top-k [30] prompts the model to consider multiple likely answers before reporting confidence. CoT [32, 33] elicits confidence after chain-of-thought reasoning. ConfTuner [19] is adapted to our decoupled setting so that only the confidence generator is trained while the answer remains fixed. SFT is a supervised finetuning baseline trained directly on the same sampling-based surrogate used in our method. Together, these baselines cover prompting-based elicitation, sampling-based estimation, and fine-tuning-based confidence calibration. Evaluation metrics. We assess verbalized confidence from two complementary perspectives: calibration and failure prediction. For calibration, we use Expected Calibration Error (ECE) [24]. For ranking-based failure prediction, we use Spearman correlation [27], Area Under the Risk– Coverage Curve (AURC) [4], and Excess AURC (EAURC) [5]. More details are in the Appendix C. Results on MMLU. Table 1 reports the calibration and failure prediction performance on MMLU across three foundation models. Overall, Ours-ORCE consistently achieves the best or near-best calibration performance while preserving the original answer accuracy. In particular, it obtains the lowest ECE on LLAMA-3 8B and Qwen3 8B, reducing ECE from 0.170 to 0.025 and from 0.212 to 0.034 compared with Vanilla, and substantially outperforming ConfTuner on both models. On Mistral 7B, Ours-ORCE achieves the strongest failure prediction performance, with the highest Spearman correlation, lowest AURC/E-AURC, and best AUPR among all methods, while maintaining the same accuracy as Vanilla. Compared with prompting-based baselines such as Top-k and CoT, Ours-ORCE provides much stronger calibration and ranking performance, suggesting that prompting alone is insufficient to reliably align verbalized confidence. Results on DROP. Table 2 reports the results on DROP. Overall, Ours-ORCE achieves the strongest confidence-alignment performance across the three foundation models while preserving the original answer-generation performance. Compared with vanilla models, Ours-ORCE substantially reduces ECE and improves Spearman correlation, AURC, E-AURC, and AUPR, indicating that the learned verbalized confidence is better aligned with correctness likelihood and more useful for failure prediction. For LLAMA-3 8B, Ours-ORCE obtains the best Spearman correlation, AURC, and E-AURC, and also improves ECE over ConfTuner and SFT. For Qwen 7B, Ours-ORCE achieves the 7
Foundation Models
Methods
LLAMA-3 8B
Vanilla Consistency Top-k CoT ConfTuner SFT Ours-ORCE
ECE ↓
Spearman Correlation
AURC ↓
E-AURC ↓
Accuracy ↑
4.682 × 10−5 0.000 1.236 × 10−70 −132 3.058 × 10 0.000 0.000 0.000
0.332 0.196 0.299 0.313 0.172 0.165 0.156
0.265 0.127 0.224 0.231 0.105 0.098 0.089
0.657 0.653 0.639 0.624 0.657 0.657 0.657
0.198 0.368 0.247 0.393 0.400 0.390 0.414
1.143 × 10−123 0.000 8.568 × 10−195 0.000 0.000 0.000 0.000
0.175 0.163 0.177 0.140 0.121 0.149 0.116
0.139 0.128 0.136 0.102 0.085 0.113 0.080
0.749 0.745 0.726 0.738 0.749 0.749 0.749
0.010 0.371 0.171 0.126 0.348 0.316 0.408
0.240 0.000 8.971 × 10−93 5.432 × 10−51 0.000 0.000 0.000
0.374 0.275 0.347 0.385 0.224 0.228 0.206
0.292 0.193 0.261 0.283 0.143 0.147 0.124
0.625 0.623 0.614 0.584 0.625 0.625 0.625
correlation ↑
p_value ↓
0.170 0.185 0.181 0.232 0.167 0.068 0.025
0.034 0.464 0.149 0.204 0.409 0.433 0.477
Qwen3 8B
Vanilla Consistency Top-k CoT ConfTuner SFT Ours-ORCE
0.212 0.204 0.043 0.137 0.088 0.130 0.034
Mistral 7B
Vanilla Consistency Top-k CoT ConfTuner SFT Ours-ORCE
0.248 0.274 0.247 0.284 0.084 0.066 0.073
Table 1: Alignment performance of methods across the foundation models on the MMLU dataset. Foundation Models
Methods
LLAMA-3 8B
Vanilla Consistency Top-k CoT ConfTuner SFT Ours-ORCE
ECE ↓
Spearman Correlation
AURC ↓
E-AURC ↓
F1 ↑
0.021 0.000 3.848 × 10−32 1.588 × 10−10 0.000 0.000 0.000
0.459 0.272 0.550 0.373 0.243 0.237 0.218
0.319 0.123 0.383 0.278 0.102 0.097 0.078
0.568 0.551 0.524 0.647 0.568 0.568 0.568
0.012 0.413 0.176 0.278 0.531 0.625 0.672
0.248 0.000 4.371 × 10−67 −169 4.132 × 10 0.000 0.000 0.000
0.472 0.312 0.349 0.199 0.275 0.226 0.214
0.328 0.180 0.245 0.161 0.131 0.082 0.070
0.568 0.579 0.626 0.777 0.568 0.568 0.568
0.064 0.523 0.136 0.185 0.081 0.481 0.578
2.931 × 10−10 0.000 7.944 × 10−41 3.743 × 10−74 2.143 × 10−15 0.000 0.000
0.469 0.312 0.467 0.343 0.506 0.305 0.274
0.300 0.129 0.283 0.241 0.336 0.135 0.104
0.526 0.506 0.505 0.628 0.526 0.526 0.526
correlation ↑
p_value ↓
0.327 0.038 0.373 0.221 0.119 0.093 0.074
-0.024 0.514 -0.120 0.065 0.571 0.568 0.628
Qwen3 8B
Vanilla Consistency Top-k CoT ConfTuner SFT Ours-ORCE
0.314 0.266 0.310 0.142 0.114 0.049 0.048
Mistral 7B
Vanilla Consistency Top-k CoT ConfTuner SFT Ours-ORCE
0.512 0.132 0.433 0.249 0.021 0.058 0.054
Table 2: Alignment performance of methods across the foundation models on the DROP dataset.
best results on nearly all confidence metrics, including the lowest ECE, highest Spearman correlation, lowest AURC/E-AURC, and highest AUPR. For Mistral 7B, Ours-ORCE achieves the best Spearman correlation and failure prediction metrics, although ConfTuner obtains the lowest ECE; this suggests that pointwise calibration alone does not necessarily imply better ranking quality or failure prediction performance. Results on ReClor. Table 3 reports results on ReClor, where Ours-ORCE consistently improves verbalized confidence alignment across all three foundation models while preserving answer accuracy. Compared with Vanilla, Ours-ORCE substantially reduces ECE from 0.178 to 0.065 on LLAMA-3 8B, from 0.164 to 0.044 on Qwen 7B, and from 0.468 to 0.061 on Mistral 7B. It also achieves the strongest Spearman correlation and the best AURC/E-AURC and AUPR across all models, indicating better ranking of correct versus incorrect responses and stronger failure prediction performance. 8
Foundation Models
Methods
LLAMA-3 8B
Vanilla Consistency Top-k CoT ConfTuner SFT Ours-ORCE
ECE ↓
Spearman Correlation
AURC ↓
E-AURC ↓
Accuracy ↑
0.239 0.110 0.177 0.134 3.196 × 10−15 8.378 × 10−26 8.061 × 10−40
0.373 0.355 0.441 0.441 0.250 0.201 0.169
0.283 0.269 0.298 0.304 0.161 0.112 0.080
0.608 0.616 0.516 0.524 0.608 0.608 0.608
0.059 0.311 0.133 0.084 0.351 0.537 0.626
0.187 1.206 × 10−12 2.797 × 10−3 0.062 6.047 × 10−16 9.431 × 10−39 8.051 × 10−56
0.190 0.147 0.161 0.183 0.091 0.083 0.064
0.167 0.126 0.139 0.161 0.069 0.060 0.041
0.794 0.802 0.800 0.796 0.794 0.794 0.794
0.049 0.029 0.153 0.124 0.191 0.344 0.392
0.275 0.523 6.216 × 10−4 0.006 1.712 × 10−5 2.368 × 10−15 8.905 × 10−20
0.477 0.489 0.438 0.389 0.408 0.329 0.306
0.332 0.346 0.292 0.271 0.263 0.184 0.161
0.512 0.514 0.510 0.556 0.512 0.512 0.512
correlation ↑
p_value ↓
0.178 0.376 0.270 0.338 0.140 0.084 0.065
0.053 0.072 0.061 0.067 0.343 0.446 0.544
Qwen3 8B
Vanilla Consistency Top-k CoT ConfTuner SFT Ours-ORCE
0.164 0.167 0.046 0.124 0.047 0.062 0.044
Mistral 7B
Vanilla Consistency Top-k CoT ConfTuner SFT Ours-ORCE
0.468 0.483 0.357 0.300 0.141 0.073 0.061
Table 3: Alignment performance of methods across the foundation models on the ReClor dataset. Foundation Models
Methods
LLAMA-3 8B
Vanilla SaySelf-inspired RL objective W/o gen (SC) Numeric dist NRD Ours-ORCE
ECE ↓ 0.170 0.038 0.143 0.137 0.110 0.025
Spearman Correlation correlation ↑
p_value ↓
0.034 0.449 0.468 0.469 0.473 0.477
4.682 × 10−5 0.000 0.000 0.000 0.000 0.000
AURC ↓
E-AURC ↓
Accuracy ↑
0.332 0.178 0.160 0.160 0.159 0.156
0.265 0.111 0.093 0.093 0.092 0.089
0.657 0.657 0.657 0.657 0.657 0.657
Table 4: Ablation study results of our method on MMLU dataset with LLAMA-3 8B. Compared with ConfTuner and SFT, Ours-ORCE further improves calibration and ranking metrics, showing the benefit of order-aware alignment beyond standard fine-tuning. Ablation study. We conduct ablation studies on MMLU with LLAMA-3 8B to evaluate the contribution of each component in our method. First, we compare against a SaySelf-inspired reward, which selects high verbalized confidence for correct responses and low verbalized confidence for incorrect responses. As shown in Table 4, this objective achieves relatively low ECE, but performs worse on ranking-based metrics such as Spearman correlation, AURC, and E-AURC, suggesting that correctness-dependent confidence preference alone is insufficient to learn fine-grained confidence ordering. Second, we replace our rank-based reward with a pointwise numeric-distance objective between verbalized confidence and the surrogate confidence score. This variant also underperforms Ours-ORCE, indicating that directly matching absolute confidence values is less effective than preserving the relative reliability ordering among responses. Third, the W/o gen (SC) variant replaces the generated verbalized confidence list used in reward construction with the surrogate confidence list. This variant performs worse on all calibration and failure-prediction metrics, indicating that effective alignment requires explicitly optimizing the relative ordering of the model’s own generated confidence outputs, rather than relying only on the surrogate ranking. Finally, we compare our Spearman correlation reward with the Numeric Ranking Difference (NRD) reward. Although NRD improves over several ablations, Ours-ORCE achieves the best ECE, Spearman correlation, AURC, and E-AURC, demonstrating that global order-aware alignment is more effective than local rankdifference matching. Notably, all variants preserve the same answer accuracy, further supporting the benefit of decoupling confidence optimization from answer generation. Table 5 evaluates the transferability of confidence-alignment methods trained on LLAMA-3 8B to other foundation models. Ours-ORCE consistently achieves the best performance on both Qwen 7B and Mistral 7B, obtaining the lowest ECE, highest Spearman correlation, and lowest AURC/E-AURC 9
Foundation Models
Methods
Qwen3 8B
Vanilla ConfTuner SFT Ours-ORCE
Mistral 7B
Vanilla ConfTuner SFT Ours-ORCE
ECE ↓
Spearman Correlation
AURC ↓
E-AURC ↓
Accuracy ↑
1.143 × 10−123 0.000 0.000 0.000
0.175 0.122 0.118 0.107
0.139 0.086 0.084 0.073
0.749 0.749 0.749 0.749
0.240 0.000 0.000 0.000
0.374 0.182 0.182 0.169
0.292 0.100 0.100 0.087
0.625 0.625 0.625 0.625
correlation ↑
p_value ↓
0.212 0.290 0.112 0.049
0.198 0.356 0.351 0.397
0.248 0.164 0.054 0.025
0.010 0.458 0.458 0.517
Table 5: Alignment performance of methods trained on LLAMA-3 8B and transferred to Qwen 7B and Mistral 7B. across all transferred settings. Compared with ConfTuner and SFT, Ours-ORCE shows stronger calibration and failure prediction performance, suggesting that the proposed order-aware alignment objective transfers more effectively across model families. Importantly, all methods preserve the same answer accuracy within each target model, indicating that the transferred confidence estimator improves uncertainty alignment without affecting the underlying answer-generation behavior.
5
Conclusion
In this work, we propose a decoupled and order-aware framework for verbalized confidence estimation in large language models. By separating answer generation from confidence estimation, our method improves uncertainty estimation while preserving the model’s answer-generation behavior. We further introduce a Spearman-based reinforcement learning objective that aligns verbalized confidence with a sampling-based surrogate of correctness likelihood, encouraging more reliable responses to receive higher stated confidence. Experiments across multiple reasoning and knowledge-intensive benchmarks show that our approach consistently improves calibration and failure prediction performance without degrading answer accuracy. These results suggest that reliable verbalized confidence requires not only calibrated confidence values, but also the correct ordering of confidence across responses with different levels of reliability.
10
References [1] Amos Azaria and Tom Mitchell. The internal state of an llm knows when it’s lying. In EMNLP (Findings), 2023. [2] Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In NAACL, 2019. [3] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In ICML, 2016. [4] Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. In NeurIPS, 2017. [5] Yonatan Geifman, Guy Uziel, and Ran El-Yaniv. Bias-reduced uncertainty estimation for deep neural classifiers. In ICLR, 2019. [6] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. In NeurIPS, 2024. [7] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In ICML, 2017. [8] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In ICLR, 2021. [9] Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 2025. [10] Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM computing surveys, 2023. [11] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. [12] Alain Jungo and Mauricio Reyes. Assessing reliability and challenges of uncertainty estimations for medical image segmentation. In MICCAI, 2019. [13] Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022. [14] Sanyam Kapoor, Nate Gruver, Manley Roberts, Katherine Collins, Arka Pal, Umang Bhatt, Adrian Weller, Samuel Dooley, Micah Goldblum, and Andrew G Wilson. Large language models must be taught to know what they don’t know. In NeurIPS, 2024. [15] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In NeurIPS, 2017. [16] Chen Li, Xiaoling Hu, and Chao Chen. Confidence estimation using unlabeled data. In ICLR, 2023. [17] Haitao Li, Junjie Chen, Jingli Yang, Qingyao Ai, Wei Jia, Youfeng Liu, Kai Lin, Yueyue Wu, Guozhi Yuan, Yiran Hu, et al. Legalagentbench: Evaluating llm agents in legal domain. In ACL, 2025. 11
[18] Shuyue S Li, Vidhisha Balachandran, Shangbin Feng, Jonathan S Ilgen, Emma Pierson, Pang W Koh, and Yulia Tsvetkov. Mediq: Question-asking llms and a benchmark for reliable interactive clinical reasoning. In NeurIPS, 2024. [19] Yibo Li, Miao Xiong, Jiaying Wu, and Bryan Hooi. Conftuner: Training large language models to express their confidence verbally. In NeurIPS, 2025. [20] Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. TMLR, 2022. [21] Hanmeng Liu, Jian Liu, Leyang Cui, Zhiyang Teng, Nan Duan, Ming Zhou, and Yue Zhang. Logiqa 2.0—an improved dataset for logical reasoning in natural language understanding. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023. [22] Amit Mandelbaum and Daphna Weinshall. Distance-based confidence score for neural network classifiers. arXiv preprint arXiv:1709.09844, 2017. [23] Jooyoung Moon, Jihyo Kim, Younghak Shin, and Sangheum Hwang. Confidence-aware learning for deep neural networks. In ICML, 2020. [24] Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In AAAI, 2015. [25] Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In CVPR, 2015. [26] Pranab Sahoo, Prabhash Meharia, Akash Ghosh, Sriparna Saha, Vinija Jain, and Aman Chadha. A comprehensive survey of hallucination in large language, image, video and audio foundation models. EMNLP (Findings), 2024. [27] C Spearman. The proof and measurement of association between two things. The American Journal of Psychology, 1904. [28] Elias Stengel-Eskin, Peter Hase, and Mohit Bansal. Lacie: Listener-aware finetuning for calibration in large language models. In NeurIPS, 2024. [29] Shuchang Tao, Liuyi Yao, Hanxing Ding, Yuexiang Xie, Qi Cao, Fei Sun, Jinyang Gao, Huawei Shen, and Bolin Ding. When to trust llms: Aligning confidence with response quality. In ACL (Findings), 2024. [30] Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In EMNLP, 2023. [31] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In ICLR, 2023. [32] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, 2022. [33] Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. In ICLR, 2024. [34] Tianyang Xu, Shujin Wu, Shizhe Diao, Xiaoze Liu, Xingyao Wang, Yangyi Chen, and Jing Gao. Sayself: Teaching llms to express confidence with self-reflective rationales. In EMNLP, 2024. [35] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. [36] Gal Yona, Roee Aharoni, and Mor Geva. Can large language models faithfully express their intrinsic uncertainty in words? In EMNLP, 2024. 12
[37] Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. Reclor: A reading comprehension dataset requiring logical reasoning. In ICLR, 2020. [38] Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probability estimates. In KDD, 2002.
13
A
Additional theoretical details
This appendix provides additional justification for the theoretical discussion in Section 3.4. Our goal is to clarify three points: (i) why the ensemble surrogate κs is a consistent approximation to the ideal prompt-level target η(X); (ii) why the Spearman-correlation (SC) reward should be interpreted as a marginal improvement in a global rank objective; and (iii) why a positive SFT warm-start is important when the reward is defined relative to an existing reference ranking. Because the practical training procedure uses a finite-sample surrogate, a drifting reference set, and DPO rather than exact ascent on a population objective, the results below characterize the idealized objective being approximated rather than the full dynamics of the practical algorithm. Throughout, X ∼ DX denotes a random prompt, Y ⋆ its ground-truth answer, and Ŷ ∼ πA (· | X) the answer produced by the answer LLM. The prompt-level target confidence is η(X) := P(Ŷ = Y ⋆ | X), and the confidence model outputs a scalar score S(X, Ŷ ) := fθ (X, Ŷ ) ∈ R. We assume that η(X) and S(X, Ŷ ) have continuous distributions, so that Spearman correlation is well-defined without ties. A.1
Consistency of the surrogate objective
We begin by justifying the use of the ensemble surrogate K 1 X κ(K) I{Yi = Y ⋆ }, s (X) := K i=1 where Y1 , . . . , YK are i.i.d. samples from πA (· | X). Proposition 1 (Pointwise consistency of the surrogate). For DX -almost every x, a.s.
κ(K) −→ η(x) s (x) −
as K → ∞.
Consequently, a.s.
κ(K) −→ η(X) s (X) − as random variables on the joint probability space. Proof. For each fixed x, the indicators I{Yi = y ⋆ } are i.i.d. Bernoulli random variables with mean P(Yi = y ⋆ | x) = η(x). The strong law of large numbers therefore gives a.s.
κ(K) −→ η(x). s (x) − Since this holds for DX -almost every x, the same conclusion holds for the random variable X. Proposition 2 (Population-level consistency of Spearman alignment). Assume the marginals of η(X) and S(X, Ŷ ) are continuous. If the population Spearman functional is continuous at the joint law of (S(X, Ŷ ), η(X)), then ρS S(X, Ŷ ), κ(K) − ρS S(X, Ŷ ), η(X) s (X) → as K → ∞. Proof sketch. By Proposition 1, a.s.
κ(K) −→ η(X). s (X) − Since S(X, Ŷ ) does not depend on K, this yields joint almost-sure convergence a.s. S(X, Ŷ ), κ(K) −→ S(X, Ŷ ), η(X) . s (X) − Hence the corresponding joint laws converge weakly. Under continuity of the population, Spearman’s functional law at the limit, the claim follows by the continuous mapping principle. Proposition 2 formalizes the sense in which optimizing SC against the surrogate becomes equivalent, as K grows, to optimizing SC against the ideal target η(X). 14
A.2
Why the SC reward is global
The SC reward used in training is ∆SC (c, κ; C, K) := rs (C ∪ {c}, K ∪ {κ}) − rs (C, K), where (C, K) is a current reference set. Unlike local rank-matching rewards, this quantity depends on how inserting (c, κ) changes the rank correlation of the entire enlarged set. This global dependence is the key reason SC can distinguish candidates with the same local rank error. Proposition 3 (SC as a marginal global objective). Let (C, K) be a finite reference set with no ties, and let (c, κ) be a candidate pair. Then ∆SC (c, κ; C, K) is exactly the marginal change in the global rank-correlation statistic rs induced by inserting the candidate. Consequently, two candidates can receive different SC rewards even when they have the same local rank error, provided they interact differently with the surrounding rank structure. Proof. The first statement follows directly from the definition of ∆SC . The second follows because the rank ordering of the enlarged set depends not only on the inserted point’s own rank, but also on how the insertion changes the relative ordering of the existing points in the augmented ranking. Therefore, SC depends on the candidate’s compatibility with the full reference structure, not only on its pointwise rank difference. Proposition 3 captures the conceptual distinction between SC and local rewards such as NRD: NRD is a pointwise rank-matching objective, whereas SC is a global rank-consistency objective. A.3
Why positive warm-start matters
The SC reward is defined relative to the current monotone trend encoded by (C, K). This makes initialization important. Proposition 4 (Sign symmetry of rank correlation). If a score function S achieves Spearman correlation ρS (S, η) = 1, then −S achieves Spearman correlation −1. More generally, rankcorrelation objectives alone do not distinguish the desired monotone solution from an anti-monotone one without an orientation condition. Proof. If S = T ◦ η for a strictly increasing T , then −S = (−T ) ◦ η with −T strictly decreasing. Therefore, the rank ordering induced by −S is exactly the reverse of that induced by η, implying Spearman correlation −1. Proposition 4 shows that an order-only objective must be oriented correctly: without some positive initial alignment, a rank-based update rule can, in principle, refine the wrong monotone trend just as consistently as the correct one. Proposition 5 (Interpretation of the warm-start condition). Suppose the SFT-initialized score S0 satisfies ρS S0 (X, Ŷ ), η(X) > 0. Then the induced confidence ranking is already positively aligned with the target ranking. In this regime, maximizing the SC marginal reward favors candidates whose insertion is more compatible with the target monotone trend than candidates that further disrupt it. Proof sketch. The SC reward measures the change in rank correlation after inserting a candidate pair. When the existing reference set is positively aligned with the target, a candidate that preserves this trend tends to increase the enlarged-set correlation, while a candidate that creates additional inversions tends to decrease it. Thus, under a positive warm-start, the SC reward acts as a local rank-refinement signal. The argument is qualitative rather than a full convergence proof because the practical procedure uses finite reference sets and a drifting optimization target. Proposition 5 is the formal role played by the SFT stage in our method: SFT is not assumed to solve the calibration problem, but it initializes the model in a regime where SC refinement is oriented toward the correct monotone structure. 15
A.4
Limitations of the idealized analysis
We close with explicit acknowledgment of the gap between the idealized population analysis and the practical DPO-based training procedure. (K)
Finite-sample surrogate. The surrogate κs (X) is a Monte Carlo estimate of η(X) and therefore introduces both variance and discretization. Our analysis characterizes the large-K objective, not the exact finite-K one. Drifting reference set. The SC reward is evaluated relative to a reference set (C, K) that evolves during training. The propositions above explain the directional role of this reward, but do not constitute a full dynamical-systems analysis of the coupled evolution of the model and the reference set. DPO approximation. The practical algorithm does not directly maximize expected SC reward. Instead, SC is used to construct preference pairs, and DPO performs a smoothed preference-optimization step. Thus, our theory should be interpreted as describing the objective that DPO is intended to approximate, rather than as a literal proof of convergence of the training algorithm. In summary, the theoretical analysis supports three claims: (i) the ensemble surrogate consistently estimates the prompt-level target η(X); (ii) the SC reward is naturally interpreted as a marginal improvement in a global rank objective; and (iii) a positive warm-start is important because it places the model in a regime where this global rank-refinement signal is oriented toward the desired monotone solution.
B
Extra results
Analysis. As shown in Table 6, for LogiQA 2.0 in distribution evaluation, our method achieves the strongest or near-strongest performance across all three foundation models, especially on the rankingbased metrics that are most closely aligned with our objective. For LLaMA-3 8B, Ours-ORCE obtains the highest Spearman correlation and the lowest AURC and E-AURC, indicating substantially better failure prediction than all baselines. On Qwen 7B, Ours-ORCE again achieves the best ranking performance and the best risk–coverage trade-off, while ConfTuner yields the lowest ECE, suggesting that our method is particularly effective at preserving the relative ordering of reliability even when absolute calibration may not always be optimal. On Mistral 7B, Ours-ORCE remains the best method on Spearman correlation, AURC, and E-AURC, although the gains over SFT and Consistency are smaller, indicating that the margin of improvement depends on the underlying model. Across all settings, these results support our central claim that optimizing a global rank-based objective is especially beneficial for failure prediction and failure prediction, while remaining competitive on calibration.
C
Experiment details
Benchmarks We conduct experiments on MMLU, DROP, LogiQA, and ReClor to evaluate different aspects of our method. MMLU [8] covers a wide range of academic and professional subjects and tests whether our method improves calibration for knowledge-intensive multiple-choice reasoning. DROP [2] requires models to perform discrete reasoning over passages, making it suitable for evaluating confidence estimation under passage-grounded numerical and compositional reasoning. For logical reasoning, we use LogiQA 2.0 [21] for training and evaluate on ReClor [37] in the main text, allowing us to assess out-of-domain generalization of the proposed order-aware objective when correctness depends on multi-step inference rather than factual recall. We report in-domain LogiQA 2.0 testing results in the Appendix B. Together, these benchmarks test whether our decoupled confidence-estimation framework generalizes across knowledge-intensive, discrete reasoning, and logical-reasoning settings while preserving answer accuracy. Evaluation metrics. We evaluate verbalized confidence from both calibration and ranking perspectives. Expected Calibration Error (ECE) [24] measures how well the model’s stated confidence 16
Foundation Models
Methods
LLAMA-3 8B
Vanilla Consistency Top-k CoT ConfTuner SFT Ours-ORCE
ECE ↓
Spearman Correlation
AURC ↓
E-AURC ↓
Accuracy ↑
0.498 1.643 × 10−32 4.708 × 10−5 1.767 × 10−16 2.718 × 10−53 4.415 × 10−61 5.169 × 10−86
0.503 0.384 0.501 0.538 0.346 0.326 0.306
0.346 0.214 0.326 0.312 0.190 0.170 0.149
0.496 0.478 0.470 0.408 0.496 0.496 0.496
0.154 0.360 0.125 0.160 0.397 0.410 0.454
7.519 × 10−10 3.390 × 10−49 6.361 × 10−7 1.792 × 10−10 1.227 × 10−60 7.000 × 10−65 1.232 × 10−80
0.251 0.210 0.257 0.243 0.151 0.161 0.148
0.200 0.162 0.206 0.197 0.101 0.110 0.098
0.699 0.706 0.697 0.710 0.699 0.699 0.699
-0.032 0.304 0.034 0.105 0.177 0.303 0.314
0.207 4.767 × 10−35 0.181 3.21 × 10−5 1.758 × 10−12 8.419 × 10−35 2.700 × 10−37
0.492 0.386 0.511 0.576 0.439 0.360 0.355
0.335 0.205 0.338 0.341 0.283 0.203 0.199
0.496 0.462 0.473 0.398 0.496 0.496 0.496
correlation ↑
p_value ↓
0.291 0.188 0.448 0.407 0.063 0.045 0.057
0.017 0.293 0.088 0.206 0.374 0.399 0.467
Qwen3 8B
Vanilla Consistency Top-k CoT ConfTuner SFT Ours-ORCE
0.250 0.220 0.195 0.126 0.039 0.094 0.081
Mistral 7B
Vanilla Consistency Top-k CoT ConfTuner SFT Ours-ORCE
0.498 0.163 0.448 0.416 0.126 0.059 0.070
Table 6: Alignment performance of methods across the foundation models on the LogiQA dataset.
matches empirical accuracy by grouping predictions into confidence bins and computing the weighted average gap between confidence and accuracy; lower ECE indicates better calibration. Spearman correlation [27] evaluates whether confidence scores correctly preserve the relative ordering of response correctness likelihood, measuring the rank correlation between verbalized confidence and correctness labels or surrogate correctness scores; higher Spearman correlation indicates better order alignment. Area Under the Risk–Coverage Curve (AURC) [4] evaluates failure prediction performance by sorting examples according to confidence and measuring the error rate as low-confidence examples are rejected; lower AURC means the model can better identify unreliable predictions. Excess AURC (EAURC) [5] subtracts the optimal AURC achievable at the same accuracy from the model’s AURC, providing an accuracy-normalized measure of failure prediction quality; lower EAURC indicates better confidence-based ranking beyond what is explained by accuracy alone. Together, these metrics assess absolute calibration, relative confidence ordering, and the usefulness of confidence for abstention. Baselines. We compare our method against several representative confidence elicitation and alignment baselines. Vanilla uses our decoupled answer-conditioned prompting strategy without any training or fine-tuning: the model first generates an answer, and then reports verbalized confidence conditioned on the fixed question–answer pair. Self-Consistency follows prior confidence elicitation work [33] by sampling multiple responses for the same question and estimating confidence from their agreement. Top-k follows the calibrated elicitation strategy of Tian et al. [30], where the model is prompted to consider multiple likely answers before reporting its confidence. CoT uses chain-of-thought [32] prompting before eliciting confidence, as studied in Xiong et al. [33], to test whether explicit reasoning improves confidence calibration. ConfTuner [19] is a fine-tuning baseline that trains LLMs to express verbalized confidence using a tokenized Brier-score objective. For a fair comparison, we adapt ConfTuner to our decoupled answer conditioned setting, so that it differs from our method only in the confidence alignment objective. Specifically, we apply the ConfTuner loss only when training the verbalized confidence generator, while keeping the generated answer fixed. During inference, the adapted ConfTuner baseline takes the question with the original model’s response as input and predicts only the verbalized confidence, without modifying the answer itself. Finally, SFT denotes a supervised fine-tuning baseline trained directly on the sampling-based correctness surrogate used in our method. This baseline isolates the effect of our order-aware reinforcement learning objective from simply fitting surrogate confidence targets. Together, these baselines cover prompting-based elicitation, sampling-based confidence estimation, and fine-tuning-based verbalized confidence calibration. 17
C.1
Compute resources
All experiments are conducted on a compute node equipped with 128 CPU cores (Intel Xeon Platinum 8462Y+) and 4 NVIDIA H100 GPUs with 80GB HBM3 memory each. C.2
Experimental setting
We implement our method using the OpenRLHF framework. We first train the supervised fine-tuning (SFT) model for 3 epochs and then further optimize it with reinforcement learning for 3 epochs. For both MMLU and DROP, we randomly sample 10,000 training examples for SFT and 20,000 training examples for RL. For LogiQA 2.0, we use one-third of the training set for SFT and the remaining two-thirds for RL.
D
Broader impact
This work aims to improve the reliability of language models by aligning verbalized confidence with answer correctness likelihood. A potential positive societal impact is that better-calibrated confidence estimates can support safer model deployment, for example by improving failure prediction, failure prediction, abstention, and routing decisions. At the same time, this work also carries potential risks. Users may over-trust verbalized confidence even when it remains imperfect, and miscalibrated confidence estimates could be especially harmful in high-stakes settings such as healthcare, law, or education if used without appropriate human oversight. We therefore view this method as a tool for improving model reliability, but not as a substitute for domain-specific safeguards or human judgment in consequential applications.
18