GUT: Quantifying and Optimizing the Reasoning Uncertainty of LLMs via Graph Complexity Shuang Liang1,2
arXiv:2609.05284v1 [cs.AI] 4 Sep 2026
1
Xin-Yu Hu1,2
Xiang-Jun Ou2
Shao-Qun Zhang1,2,B
National Key Laboratory for Novel Software Technology, Nanjing University, China. 2 School of Intelligent Science and Technology, Nanjing University, China. [email protected]
Abstract Recent years have witnessed great advances in the reasoning ability of Large Language Models (LLMs). However, the reasoning processes of LLMs often exhibit uncertainty, where LLMs often produce a proliferation of divergent branches at each reasoning step even when fed the same prompting inputs, and certain branches exhibit evidently incredible, even nonsensical, reasoning chains and results. In this paper, we propose the Graph-complexity-based UncerTainty (GUT) method for investigating the reasoning uncertainty of LLMs. The key idea of GUT is to characterize the potential branches of each reasoning chain with a directed acyclic graph, thereby ensuring that all potential branches are comprehensively covered within the graph space. Building upon this recognition, we further build two modules of GUT, that is, a Quantification (GUT-Q) module and an Optimization (GUT-O) module, for quantifying and reducing the reasoning uncertainty of LLMs, respectively. GUT-Q measures LLM reasoning uncertainty by approximating the reasoning space complexity with graph complexity. GUT-O implements uncertainty optimization by treating negative uncertainty as the reward function in reinforcement learning. Experimental results conducted on four LLMs and five datasets validate the effectiveness of GUT. Key words: Large Language Models, Reasoning Uncertainty Quantification, Reasoning Uncertainty Optimization, Graph Complexity
1. Introduction
Uncertainty arises when the intrinsic stochasticity of the system results in the variability in its outputs or decisions [33]. In particular, when developers employ Large Language Models (LLMs) for reasoning, they frequently encounter the reasoning uncertainty originating from temperature-based stochastic sampling over token probability distributions, which in turn induces variability in the generated reasoning chains [24]. Thus, it is necessary and significant to quantify and optimize the reasoning uncertainty Preprint
September 7, 2026
Figure 1: Illustration of our proposed GUT. A real-world case study of GUT is provided in Appendix E. of LLMs, which has been widely deployed in fields of medical care [1], autonomous driving [44], and quantitative trading [29]. There have been lots of efforts on quantifying the reasoning uncertainty of LLMs. An intuitive way is to prompt the LLM to judge its own reasoning uncertainty [20, 47]. However, the effectiveness of this manner is fundamentally limited by the inherent stochasticity of the prompt-based generation process, induced by the token-level distribution and stochastic sampling, thus recursively amplifying the uncertainty in LLM reasoning. Subsequent studies modeled the reasoning chain as a sequence, based on which researchers quantified the reasoning uncertainty of LLMs by exploiting either the statistical information of a single sampled sequence [22, 25] or the diversity of multiple sampled sequences [12, 23]. Nevertheless, modeling reasoning chains as simple sequences fails to accurately capture the inherent uncertainty in LLM reasoning, as it overlooks the potential branches emerging at each individual reasoning step. Recent studies [8, 27, 54] constructed graphs to explicitly model potential branches of reasoning steps and quantified the reasoning uncertainty by the topological information of graphs. The graph-based methods achieve a considerably precise modeling of reasoning branches, but rely heavily on self-prompting. Therefore, achieving a precise and effective characterization of the reasoning branches induced by both token-level distributions and stochastic sampling is necessary and challenging for quantifying the LLM reasoning uncertainty. We implement graph modeling by performing multiple sampling and merging equivalent reasoning steps with a Natural Language Inference (NLI) model, thereby precisely characterizing potential branches in each reasoning step and overcoming effectiveness bottlenecks of self-prompting. Since the collection of all potential branches, that is, the reasoning space [5], covers all possible reasoning chains induced by stochastic sampling, we can exploit the graph complexity to approximate reasoning space complexity. 2
We construct the graph complexity by integrating statistics over token-level distributions and topological information. The constructed graph complexity is a more comprehensive estimator for quantifying LLM reasoning uncertainty than those of existing UQ methods, since one can trace the reasoning uncertainty back to the stochasticity in both token-level distributions and sampling. By taking the graph-complexity-based uncertainty with a simple proxy approximation as a reward function, we can build the optimization problem within Reinforcement Learning (RL) for reducing the LLM reasoning uncertainty. Figure 1 illustrates our key ideas of quantifying and optimizing the reasoning uncertainty of LLMs. Based on the above recognition, we propose the Graph-complexity-based UncerTainty (GUT) for quantifying and reducing the reasoning uncertainty, comprising a Quantification (GUT-Q) module and an Optimization (GUT-O) module. Our contributions are summarized as follows. Firstly, we propose the GUT-Q for UQ in LLM reasoning that operates by estimating the reasoning space complexity with graph complexity. Empirical evidence across 4 LLMs and 5 datasets shows that the GUT-Q outperforms 45 UQ contenders in the downstream task of selective generation, measured by AUROC, AUPRC, and PRR. Secondly, we propose the GUT-O for Uncertainty Optimization (UO) in LLM reasoning that operates by setting the negative uncertainty proxy as the reward in the RL framework. Empirical evidence across 4 LLM scales and 5 datasets shows that GUT-O effectively reduces the reasoning uncertainty and improves the accuracy. The rest of this paper is organized as follows. Section 2 reviews related work. Section 3 and Section 4 formally introduce the GUT-Q and GUT-O modules, respectively. Section 5 conducts experiments to validate the effectiveness of our proposed GUT. Section 6 concludes this work.
2. Related Work
Reasoning Uncertainty Quantification. Intuitively, one can prompt the LLM to judge its own reasoning uncertainty with sophisticated templates such as multiple choice-based prompts [20] and reasoning decomposition-based prompts [47]. The prompt-based generation process exhibits inherent stochasticity induced by token-level distribution and stochastic sampling, which recursively amplifies the uncertainty in LLM reasoning. Thus, self-prompting manners are fundamentally limited in their effectiveness. By modeling the reasoning chain as a sequence, subsequent studies quantified the reasoning uncertainty of LLMs through the exploitation of either the statistical information of a single sampled sequence like entropy [25] and likelihood [22], or the diversity among multiple sampled sequences such as eccentricity [23] and the entropy of semantic clusters [12]. These methods depend heavily on the statistical and information-theoretic measures upon the sequence-level modeling. Nevertheless, modeling 3
reasoning chains merely as sequences fails to accurately capture the inherent uncertainty in LLM reasoning, since it neglects the potential branches emerging at each reasoning step. A natural approach to characterize potential branches at each reasoning step is to use a graph to explicitly model the reasoning space, defined as the set of all possible reasoning chains of a problem [5]. Recent studies constructed graphs and quantified reasoning uncertainty by the topological information of graphs, such as graph distance [8] and the number of paths [54]. Although the graph-based methods achieve a considerably precise modeling of reasoning branches, they rely heavily on self-prompting. Therefore, it remains necessary and challenging to achieve a precise and effective characterization of the reasoning branches induced by token distributions and stochastic sampling for quantifying the reasoning uncertainty of LLMs.
Reasoning Uncertainty Optimization. There has been a surge of interest in incorporating uncertainty measures to enhance LLM reasoning. Bi et al. [2] utilized LLM logits to identify and prevent the expansion of high-uncertainty nodes in the ToT model [49]. Some studies detected and terminated the generation of highly uncertain reasoning chains by leveraging the statistics of token-level distributions, such as entropy [14], the probability gap between the top-1 and top-2 tokens [57], and the logit variation between consecutive token positions [51]. Hence, these studies primarily focused on exploring the reasoning space by exploiting uncertainty as a signal, rather than reducing the reasoning uncertainty. Therefore, the UO in LLM reasoning remains largely unexplored.
3. Uncertainty Quantification
In this section, we formally propose the GUT-Q module for UQ in LLM reasoning. Before that, we introduce some useful notations. Let [N ] = {1, 2, ..., N } be an integer set for N ∈ N+ , and | · | denotes the number of elements in a collection, e.g., |[N ]| = N . Let ⌊z⌋ denote the greatest integer less than or equal to z ∈ R, e.g., ⌊1.1⌋ = 1. We start by formalizing the process of LLM reasoning. Given a problem x sampled from the distribution X and an instruction I, the LLM parameterized by θ maps the context (x, I) to a token-level distribution πθ over the vocabulary V. A single reasoning process typically takes the form of a Chain-of-Thought (CoT) ci , containing a sequence of ni ∈ N+ reasoning steps {sik }k∈[ni ] for i ∈ N+ . At each step, the LLM performs temperature sampling over the token-level distribution πθ (·|h<j ) autoregressively to obtain a sequence of tokens and form a new context, where h<j denotes the context (tj−1 , ..., t1 , x, I) for 2 ≤ j ≤ |c| and (x, I) for j = 1. Hence, the stochasticity of temperature sampling over the token-level distribution πθ (·|h<j ) induces nk ∈ N+ potential branches {sik }i∈[nk ] at the k-th step for k ∈ [ni ]. The CoT ci is generated by recursively repeating this sampling process. Finally, one can obtain the 4
reasoning chain ci = {sik }k∈[ni ] , where the last step sini contains the final answer. This final answer admits potential candidates induced by the potential branches at each step. As shown in Figure 1, after merging semantically equivalent nodes, for the LLM with parameter θ and flat token-level distributions, there are four potential branches at the first reasoning step and two final answers; while for the LLM with parameter θ′ and peak token-level distributions, there are only two potential branches at the first step and one final answer. It is observed that more potential branches at each step would induce a more complex topology of the underlying reasoning space, which can be characterized as higher complexity. Based on this observation, it is natural to quantify the reasoning uncertainty by the complexity of the reasoning space, which can be approximated by the graph complexity. The proposed GUT-Q constructs a graph to reveal the potential branches of each step, then quantifies step-level uncertainty to capture token-level stochasticity, and finally exploits graph complexity to approximate reasoning space complexity, which in turn enables the estimation of reasoning uncertainty. GUT-Q comprises graph construction, step-level uncertainty quantification, and graph complexity estimation, detailed in Subsections 3.1, 3.2, and 3.3, respectively.
3.1. Graph Construction This subsection formalizes how GUT-Q constructs a Directed Acyclic Graph (DAG) to characterize the reasoning space. The key idea is to first initialize a DAG with multiple CoT chains and then merge semantically equivalent nodes to uncover the intrinsic potential branches of each step within the reasoning space. We begin by sampling K ∈ N+ reasoning chains {ci }i∈[K] and extracting reasoning steps {sij }j∈[ni ] for ni ∈ N+ . Next, we initialize a DAG rooted at the problem x. This DAG comprises nodes representing reasoning steps sij . These nodes are connected by directed edges that point to the next step. Specifically, directed edges connect the root x to the initial steps si1 and link subsequent steps sij−1 to sij . Each path terminates at a leaf node representing the final answer sini , which is typically a real-valued scalar or vector for mathematical datasets like MATH-500. Intuitively, there are three ways to merge semantically equivalent nodes. First, one can prompt an LLM to judge the equivalence of two nodes. This method suffers from high LLM inference costs and the inherent stochasticity of prompt-based generation. Second, equivalence can be established when the cosine similarity of the two nodes’ corresponding embeddings exceeds a pre-specified threshold. This method may require a task-specific choice of embedding model and careful tuning of this threshold. Third, two nodes are considered equivalent if they are bidirectionally entailed by an NLI model, i.e., a node entails the other and vice versa. This approach overcomes the limitations of the previous 5
two methods by running a small-scale NLI model that yields deterministic entailment judgments. Thus, as an example in this work, we adopt the third implementation and validate its effectiveness in Subsection 5.1 and Appendix B. When two nodes are merged, the directed edges are redirected to the newly formed node. Notably, we traverse the node pairs in the AOV-based order [19] to ensure that the graph remains a DAG after merging, a property that a naive traversal over all node pairs cannot preserve. Figure 1 illustrates the constructed DAG G = (V, E). Prompt template, pseudocode of DAG construction procedure, and configurations of hyperparameters are provided in Appendices A.1, A.2, and B.2, respectively.
3.2. Step-level Uncertainty Quantification This subsection introduces step-level uncertainty quantification for capturing stochasticity originating from token-level distributions. Let v ∈ V be an arbitrary node that represents a reasoning step s consisting of tokens {tj }j∈[|s|] . The key idea is to calculate token-level uncertainty and then aggregate them to derive step-level uncertainty. We employ four kinds of token-level uncertainty measures U (tj ) as follows. • Neg Max Prob U (tj ) = − maxv∈V πθ (v|h<j ) , • Avg Log Prob U (tj ) = • Entropy U (tj ) = −
P
P
v∈V log πθ (v|h<j )/|V|
,
v∈V πθ (v|h<j ) logπθ (v|h<j )
,
• Neg Token Prob U (tj ) = −πθ (tj |h<j ) , where Neg, Prob, and Avg denote negative, probability, and average, respectively. A larger U (tj ) indicates Pi+w−1 higher uncertainty of token tj . Next, we perform group-level aggregation as Ui = j=i U (tj )/w, yielding a set {Ui }i∈[m−w+1] for w ∈ [m]. This approach aims to capture local uncertainty in reasoning steps, which a naive average of U (tj ) over the entire step [13, 26] fails to capture [14]. Prior work indicated that certain tokens, such as the initial [57] and final ones [43], are vital to the uncertainty quantification of a reasoning step. Thus, we derive the step-level uncertainty U (s) by averaging the token-level uncertainties at specific positions. Specifically, we consider the following three ways to select these positions. • Top-d%. Selecting the largest d% in {Ui }i∈[m−w+1] . • Head-d%. Selecting the initial d% in {Ui }i∈[m−w+1] , namely {Ui }i∈[⌊(m−w+1)d%⌋] . • Tail-d%. Selecting the final d% in {Ui }i∈[m−w+1] , namely {Ui }i∈{m−1−⌊(m−w+1)d%⌋,...,m} . 6
Algorithm 1 GUT-Q-W Input: DAG G = (V, E), node uncertainty {U (v)}v∈V Output: Uncertainty-weighted width Wu Procedures: 1: L ← TopologicalSort(G) {Get topological order} 2: R[v] ← 0 for all v ∈ V {Initialize topological level} 3: for u ∈ L do 4:
R[u] ← max({R[p] + 1 | (p, u) ∈ E} ∪ {0})
5: end for 6: C ← CountFrequencies(R) 7: rw ← min({r | C[r] = max(C)}) 8: Vw ← {v | R[v] = rw } {Get width-path nodes} 9: Wu ←
P
v∈Vw U (v)
The step-level uncertainty U (s) can be viewed as an attribute of its corresponding node, termed node uncertainty U (v).
3.3. Graph Complexity Estimation In this subsection, we construct graph complexity by exploiting node uncertainty and graph topology to quantify reasoning uncertainty, as illustrated in Figure 1. Specifically, we introduce three graph complexity estimation methods, including Width (GUT-Q-W), Height (GUT-Q-H), and Uncertainty Propagation (GUT-Q-UP).
GUT-Q-W. It is intuitive to first find the nodes on the width path and then sum the node uncertainty over these nodes. The width of a DAG represents the maximum number of branches at any single step in the reasoning space. A large width indicates the existence of at least one step with numerous potential branches. Thus, the sum of node uncertainty over the width path, that is, uncertainty-weighted width Wu , can reflect the divergence of potential reasoning branches. The key idea for finding the width is to first perform topological sorting [7] to obtain the topological order of the nodes and then find the maximum number of nodes at the same topological level. Algorithm 1 summarizes the calculation process of Wu for estimating graph complexity.
GUT-Q-H. Dual to GUT-Q-W, one can find the nodes on the height path and then sum the node uncertainty over them. The height of a DAG represents the length of the longest reasoning chain from the root to a leaf node, measured by the number of edges along this path. A large height indicates the 7
Algorithm 2 GUT-Q-UP Input: DAG G = (V, E), node uncertainty {U (v)}v∈V , weight ω ∈ R+ , activation function ϕ Output: Auxiliary node uncertainty U (sa ) Procedures: 1: Construct augmented graph G′ = (V ′ , E ′ ) by connecting all leaf nodes of G to a new auxiliary node
sa 2: Construct an FNN corresponding to G′ with ϕ 3: Set weights wuv ← ω for all (u, v) ∈ E ′ 4: Set biases bv ← U (v) for all v ∈ V , and bsa ← 0 5: U (sa ) ← output of sa from forward propagation
existence of at least one reasoning chain with numerous steps. Hence, the sum of node uncertainty over the height path, namely uncertainty-weighted height, captures both the depth of the reasoning space and the token-level stochasticity along this path. The key idea of finding the height is to first perform topological sorting to obtain the topological order of the nodes, identify the maximum topological level, and then select a sequence of nodes across these levels that are connected by edges. Algorithm 3 in Appendix A.2 lists the procedure of GUT-Q-H for graph complexity estimation.
GUT-Q-UP. GUT-Q-W and GUT-Q-H may not fully exploit node uncertainty U (v) or the detailed connectivity among nodes. Thus, we propose GUT-Q-UP to make fuller use of them. Since a DAG is naturally a Feedforward Neural Network (FNN) [35] and uncertainty accumulates along reasoning steps [15, 55], it is intuitive to propagate node uncertainty U (v) from the first reasoning steps {si1 }i∈[K] to an auxiliary node sa , where all final-answer nodes {sini }i∈[K] point to it. Therefore, U (sa ), the output of this FNN after forward propagation, incorporates node uncertainty propagated throughout the reasoning space. Algorithm 2 outlines the GUT-Q-UP algorithm to estimate graph complexity, where the node uncertainty U (v) is used to initialize the biases of the FNN associated with G. Algorithm 4 in Appendix A.2 lists a variant that initializes this FNN’s weights with U (v). GUT-Q-W and GUT-Q-H excel in explainability over GUT-Q-UP, since they characterize the intrinsic topology of the reasoning space; GUT-Q-W captures the maximum number of potential branches at any single step, and GUT-Q-H integrates the length of the longest reasoning chain. In contrast, GUT-Q-UP typically outperforms GUT-Q-W and GUT-Q-H in the downstream task of selective generation [34], as shown by the empirical evidence in Subsection 5.1. These findings reveal the usage scenarios of these three graph complexity estimation methods, where one may employ the GUT-Q-W and GUT-Q-H for the explainability of the intrinsic topology of the reasoning space, and employ the GUT-Q-UP for the downstream task of selective generation.
8
4. Uncertainty Optimization
This section formally proposes the GUT-O module for optimizing the LLM reasoning uncertainty U (x) measured by GUT-Q. Intuitively, one can treat U (x) as an optimization target and solve this optimization by exploiting standard gradient-based algorithms. Unfortunately, this way is infeasible, as U (x), with the discrete nature caused by node merging, is non-differentiable with respect to LLM parameters. We adopt PN indirect optimization by exploiting the law of large numbers [3] as U (x) = Ec [U (x, c)] ≈ i=1 U (x, ci )/N for N ∈ N+ . Thus, optimizing U (x, c) helps optimize U (x), enabling us to employ a differentiable U (x, c) as an optimization target for indirect optimization. In this work, we take Mean Token Log Probability (MTLP) [26] as an example of U (x, c), which is a differentiable function over the token-level distribution and is thus directly differentiable with respect to the LLM parameters. To verify the validity of the MTLP optimization target, we calculate the Pearson PN Correlation Coefficient (PCC) between U (x) and i=1 U (x, ci )/N , where a positive PCC indicates that optimizing U (x, c) can effectively optimize U (x). Table 1 reports these PCCs on the investigated mathematical reasoning datasets, averaged across four Qwen3 LLMs. It is observed that PCCs are consistently positive across all datasets, which validates the effectiveness of the MTLP optimization target. Inspired by Prabhudesai et al. [31] and Zhao et al. [56], we employ the gradient-based GRPO algorithm [36] to optimize the MTLP objective U (x, c) by setting its negative as the reward. Details regarding PCC results of individual LLMs and the GUT-O formulation are provided in Appendices C.3 and A.3, respectively. UQ
GSM8K
MATH-500
AMC2022-2024
GUT-Q-W
0.1909 ± 0.0207
0.2364 ± 0.0315
0.3603 ± 0.0592
GUT-Q-H
0.1365 ± 0.0229
0.1945 ± 0.0318
0.3197 ± 0.0602
GUT-Q-UP
0.2110 ± 0.0213
0.2069 ± 0.0334
0.2586 ± 0.0600
Table 1: PCCs between MTLP and complexity-based uncertainty metrics across investigated mathematical reasoning datasets, averaged across 4 Qwen3 LLMs.
5. Experiments
In this section, we conduct experiments to validate the proposed GUT framework. The experiments are performed to answer (Q1) whether and to what extent the proposed UQ can contribute to discriminating between correct and incorrect generations over classical UQ methods, and (Q2) whether and to what extent the proposed GUT-O reduces the graph-complexity-based uncertainty. 9
Experiments were conducted on NVIDIA RTX PRO 6000 96G GPUs (×8). Evaluated LLMs involve the Qwen3 [48] family across scales of 8B, 4B, 1.7B, and 0.6B. The configuration of sampling parameters is detailed in Appendix B.2. We follow Farquhar et al. [12] and use the Deberta-large [17] that is fine-tuned on the MNLI [46] dataset as the NLI model. The evaluated datasets span three task types, including mathematical reasoning, first-order logic reasoning, and long-form Question Answering (QA). For mathematical reasoning, we select three datasets at increasing levels of difficulty, namely GSM8K [6], MATH-500 [21], and AMC2022-2024. For first-order logic reasoning, we use FOLIO [16]. For long-form QA, we adopt MMLU-Pro [45]. Details on the evaluated datasets are provided in Appendix B.1.
5.1. Verifications on GUT-Q This subsection validates the effectiveness of the proposed GUT-Q module. Before presenting the results, we introduce the UQ performance evaluations and contenders. Following prior work [23, 12], we use the performance on the downstream task of selective generation [34] to evaluate a UQ method, i.e., measure a UQ method’s ability to discriminate between correct and incorrect generations. We use three metrics to quantify this ability, including the Area Under the Receiver Operating Characteristic curve (AUROC), the Area Under the Precision-Recall Curve (AUPRC), and the Prediction Rejection Ratio (PRR) [40], which is a normalized version of the Area Under the Accuracy-Rejection Curve (AUARC) [23, 28]. Higher values indicate better UQ performance for all three metrics. Following Vashurin et al. [40], we bootstrap datasets 1000 times and report the mean and standard deviation of the metrics as mean±std. We compared 45 white-box and black-box UQ contenders across five categories, including the diversitybased methods like semantic entropy [12] and graph Laplacian [23], the information-based methods such as CSL [22], the self-reflexive methods like P(True) [20], the reasoning-enhanced methods like CoT-UQ [52], and the graph-based methods like TopoUQ [8]. Details are provided in Appendix B.3. Table 2 shows the comparisons of UQ performance for Qwen3-4B across partial datasets, where the upper group lists the performance of contenders and the lower group details that of ours. Bold and underline denote the best and second-best results within each group, respectively. Results across five datasets are presented in Table 10 in Appendix B.5. It is observed that the GUT-Q implemented with the UP algorithm consistently outperforms all contenders across three evaluation metrics and datasets spanning three difficulty levels. There are similar observations for all 4 investigated LLMs and 5 datasets, as detailed in Appendix B.5. These observations show that GUT-Q outperforms the best UQ contenders by an average of 11.79% in PRR, 13.33% in AUROC, and 9.66% in AUPRC, across four evaluated LLMs and five datasets, which answers Q1. Moreover, GUT-Q exhibits comparable runtime complexity to existing start-of-the-art methods such as SE and Eig-C, where Appendix B.4 provides a detailed analysis. 10
GSM8K
MATH-500
AMC2022-2024
UQ PRR
AUROC
AUPRC
PRR
AUROC
AUPRC
PRR
AUROC
AUPRC
MSP
-33.66±1.22
58.89±0.97
86.84±0.80
-8.09±2.46
58.76±1.67
60.85±2.04 -37.65±4.97
67.35±3.70
52.35±4.27
Ppl
-33.66±1.22
58.89±0.98
86.84±0.79
-8.09±2.46
58.76±1.65
60.85±2.01 -37.65±4.97
67.35±3.77
52.35±4.41
MTE
47.45±0.86
28.89±1.73
74.56±1.26
31.08±2.36
36.22±1.99
44.79±2.13
37.77±4.71
26.06±3.79
28.18±3.57
MTLP
-27.77±1.13
57.30±1.52
84.56±0.91
2.10±2.40
55.29±1.95
62.99±2.53
27.33±5.00
28.86±3.86
26.17±3.20
MPMI
-28.86±1.13
59.13±1.58
87.55±0.86
4.67±2.37
57.56±2.01
63.94±2.56 -12.39±4.79
48.12±4.37
43.94±5.05
CPMI
-11.23±0.99
43.75±1.61
80.60±1.17
15.15±2.32
47.15±1.99
57.15±2.37 -56.69±4.14
74.24±4.09
61.25±5.55
RD
14.53±1.08
41.01±0.97
83.88±0.89
25.17±2.50
41.30±1.55
55.70±1.97
41.40±5.15
33.13±3.65
39.16±4.02
FRD
14.52±1.08
41.01±0.99
83.88±0.83
25.78±2.50
41.35±1.61
55.79±1.94
41.38±5.13
33.16±3.65
39.18±4.33
AS
-17.30±1.19
53.34±1.72
83.69±0.97
18.19±2.45
48.81±2.07
57.56±2.42
25.50±4.85
29.96±3.98
30.58±3.69
CSL
-39.67±1.17
60.61±1.66
87.09±0.88
4.33±2.40
55.39±2.01
60.42±2.51
13.68±4.96
30.56±3.99
28.06±3.56
MCSE
6.55±1.11
50.00±0.00
84.31±0.79
-4.91±2.52
51.33±0.41
55.47±1.75 -26.20±4.61
52.74±1.12
41.03±3.58
MCNSE
6.55±1.11
50.00±0.00
84.31±0.81
-4.91±2.52
51.33±0.40
55.47±1.70 -26.20±4.61
52.74±1.06
41.03±3.70
RAUQ
-29.39±1.18
55.62±1.77
84.21±0.99
3.25±2.47
55.51±2.03
62.72±2.56
32.58±4.96
24.12±3.90
24.97±3.28
RAUQ-E
-40.55±1.22
60.84±1.60
85.45±0.88
5.56±2.45
55.45±2.09
62.28±2.60
39.79±4.90
22.54±3.64
24.80±3.28
SE
-6.65±1.09
52.71±1.18
85.13±0.79
0.14±2.42
56.45±1.93
61.98±2.32
0.54±4.92
55.48±4.29
50.38±5.25
SAR
59.47±0.57
26.42±1.39
75.20±1.28
33.61±2.43
35.38±1.92
44.61±1.95
51.98±4.70
26.63±3.55
28.42±3.61
TSAR
15.14±1.08
40.38±1.04
82.05±0.86
24.77±2.50
37.47±1.65
49.21±1.71
19.41±5.17
30.59±3.38
35.57±3.68
SSAR
-27.96±1.17 65.41±1.45 90.38±0.73
10.34±2.40
51.54±1.97
61.69±2.39
6.27±5.13
52.81±4.60
55.40±5.35
SD
33.10±0.78
37.76±1.49
79.16±1.15
35.85±2.39
38.87±1.99
51.82±2.37 -56.66±3.58
71.11±3.86
61.06±5.18
ES
40.13±0.78
36.16±1.55
79.43±1.20
35.57±2.45
29.81±1.82
40.59±1.83
65.02±4.28
21.69±3.66
31.01±3.66
CoCoA-MSP
-30.10±1.20
56.65±1.70
83.73±0.92
17.21±2.47
52.97±2.04
62.26±2.48
46.54±4.48
21.17±3.68
25.03±3.23
CoCoA-Ppl
-27.79±1.15
56.25±1.71
83.04±0.97
9.62±2.48
54.15±2.01
61.35±2.55
41.29±4.89
22.42±3.84
25.15±3.28
CoCoA-MTE
-14.85±1.09
51.17±1.75
81.48±1.05
12.23±2.47
53.71±2.08
61.89±2.42
43.05±4.88
21.68±3.91
24.69±3.39
P(True)
13.30±1.11
38.54±1.87
76.44±1.20
26.47±2.49
41.07±2.02
49.49±2.21
21.78±4.86
27.79±4.44
25.47±4.96
CoT-UQ-ME
14.47±5.58
55.51±1.85
87.51±1.03
33.48±4.82 63.45±1.94 69.40±2.43
25.78±9.85
61.27±4.54
51.27±5.95
CoT-UQ-MI
17.07±5.68
56.76±1.77
87.90±1.03
28.22±5.11
59.23±4.51
48.81±6.23
CoT-UQ-SAR
14.11±5.65
55.60±1.77
87.49±1.05
NS
5.15±1.10
48.00±0.89
83.66±0.84
Eig-E
46.57±0.68
Eig-C
59.48±0.64
Eig-J
61.24±1.98
67.40±2.49
20.98±9.99
33.21±4.80
63.32±1.90
69.28±2.40
26.38±9.91
61.74±4.62
51.53±5.97
4.12±2.48
50.00±0.00
54.80±1.80 -19.38±5.05
50.00±0.00
39.67±3.57
33.01±1.61
77.54±1.31 -30.53±2.42
59.51±1.92
59.90±2.51 -60.23±4.79 76.71±4.02
62.31±5.61
26.08±1.43
74.93±1.27
2.81±2.51
50.61±2.00
56.19±2.44 -54.14±4.79
73.40±3.98
61.59±5.23
42.17±0.72
33.30±1.62
76.53±1.26
50.42±2.17
28.84±1.73
42.30±1.81
21.35±3.85
28.86±3.97
Deg-E
47.13±0.66
33.16±1.46
77.73±1.27
3.98±2.42
52.22±2.05
56.64±2.54 -59.13±4.09
76.03±3.74 64.78±5.59
Deg-C
60.76±0.63
25.42±1.22
74.77±1.24
47.95±2.03
34.47±1.98
47.35±2.33 -11.14±4.93
57.42±4.48
54.36±5.05
Deg-J
43.13±0.72
32.69±1.59
76.33±1.27 50.95±2.16
28.38±1.75
41.99±1.83
58.55±4.34
21.15±4.05
28.84±4.01
Ecc-E
44.92±0.73
33.20±1.66
76.98±1.29 -11.87±2.53
53.81±1.96
56.34±2.47 -55.52±4.07
73.72±3.85
59.81±5.51
Ecc-C
56.84±0.63
27.78±1.44
75.36±1.24
32.90±2.10
43.17±2.04
53.03±2.25 -28.36±4.20
63.76±4.19
53.88±5.28
58.01±4.40
Ecc-J
44.89±0.76
30.82±1.77
75.43±1.21
49.68±2.25
27.01±1.74
41.30±1.77 67.56±4.39
17.51±3.70
26.43±3.30
LS-R1
42.64±0.70
33.21±1.60
76.48±1.22
2.72±2.54
48.62±2.06
53.11±2.23
-0.49±4.89
46.63±4.10
36.83±3.78
LS-R2
49.43±0.67
29.25±1.62
75.12±1.22
26.35±2.42
38.13±2.02
46.66±2.02
48.92±4.48
27.51±3.68
28.95±3.37
LS-RL
49.02±0.66
29.30±1.58
74.88±1.30
35.65±2.34
33.00±1.87
43.60±1.79
60.71±4.17
20.80±3.46
27.15±3.15
LS-B
45.10±0.71
31.05±1.64
75.86±1.25
29.07±2.40
37.20±1.92
45.89±2.15
47.23±4.57
27.88±3.71
28.97±3.76
KLE
-9.26±1.11
52.32±1.75
84.98±0.99
6.65±2.47
48.55±1.94
53.49±2.31
-8.96±4.93
52.03±4.21
42.45±4.45
LUQ
53.52±0.68
28.97±1.40
75.76±1.23
41.59±2.06
38.59±1.87
49.99±2.44 -25.97±4.64
62.50±4.32
56.86±5.22
CoT-UQ-SP
-3.37±6.33
51.38±1.90
85.11±1.12 -10.00±5.34
48.60±2.06
54.18±2.26
-5.71±9.67
47.69±4.60
36.71±4.63
Topo-UQ
41.75±3.74
64.77±1.59
89.94±0.79
53.01±2.20
65.75±2.58
24.04±9.16
56.96±5.37
51.65±5.64
11.12±4.69
GUT-Q-W
72.84±2.61
81.68±1.29
95.46±0.49
54.59±4.13
76.64±1.71
75.92±2.29
69.23±5.69
82.38±3.06
76.85±4.09
GUT-Q-H
55.87±3.04
71.38±1.45
92.73±0.63
46.49±4.35
72.65±1.86
72.29±2.48
59.56±6.94
78.41±3.42
69.97±5.10
GUT-Q-UP
81.68±1.83 86.61±1.02 96.93±0.36 61.93±3.59 80.13±1.57 79.45±2.07 80.82±4.24 89.48±2.19 84.60±3.43
Table 2: Comparisons of UQ performance for Qwen3-4B across partial contenders and datasets, where bold and underline denote the group best and second-best results, respectively. 11
(a) Ablation comparisons of UQ performance.
(b) Impact of sample number K on UQ performance.
(c) Impact of temperature T on UQ performance.
(d) Impact of shot number F on UQ performance.
Figure 2: Ablation and sensitivity analyses of the GUT-Q for Qwen3-4B.
Ablation Analyses. This paragraph analyzes how node uncertainty calculation and node merging affect the performance of GUT-Q. Figure 2a shows the ablation comparisons of UQ performance for Qwen3-4B across mathematical reasoning datasets. Results for all 4 concerned LLMs and 5 datasets are presented in Appendix D.1. It is observed that, compared to the blue bars, the red bars are closer while the green bars are far shorter for the UP methods, demonstrating that node merging is more important than node uncertainty in the UP algorithm. We also observe that the red and green bars are notably shorter than the blue bars for both the GUT-Q-W and GUT-Q-H, indicating that both node uncertainty and node merging are essential for GUT-Q-W and GUT-Q-H.
Sensitivity Analyses. This paragraph analyzes how UQ performance varies with the number of samples K, temperature T , and number of shots F . Figure 2b illustrates the impact of the number of samples K on UQ performance. We recommend K = 9 to balance efficiency and performance since a larger K leads to higher computational costs. Figures 2c and 2d illustrate the impact of the temperature T and 12
the number of shots F on UQ performance, respectively. We recommend T = 1.0 and F = 5 based on UQ performance. Appendix D.2 provides complete sensitivity analyses for all four evaluated LLMs across five concerned datasets, as well as for the node merging criterion in Algorithm 5.
5.2. Verifications on GUT-O This subsection validates the effectiveness of the proposed GUT-O module by comparing the mean P uncertainty [50] of an LLM on a dataset, defined as U (D; θ) = x∈D U (x; θ)/|D|, before and after using GUT-O. A greater reduction in GUT-Q-derived mean uncertainty indicates better UO performance. We use GRPO as a contender. Mean Uncertainty Models
Accuracy
MTLP GUT-Q-W
GUT-Q-H
GUT-Q-UP
Qwen3-0.6B
18.00 ± 1.31
−7.55 ± 0.04
0.77 ± 0.01
1.86 ± 0.03
1.14 ± 0.01
+ GRPO
23.40 ± 1.51
−7.67 ± 0.04
0.74 ± 0.01 (-3.9%)
1.88 ± 0.03 (+1.1%)
1.17 ± 0.02 (+2.6%)
+ GUT-O
21.40 ± 1.43
−8.94 ± 0.04
0.64 ± 0.01 (-16.9%)
1.51 ± 0.03 (-18.8%)
0.88 ± 0.01 (-22.8%)
Qwen3-1.7B
42.20 ± 1.79
−13.45 ± 0.06
0.43 ± 0.01
2.03 ± 0.05
0.47 ± 0.01
+ GRPO
46.40 ± 1.81
−13.56 ± 0.07
0.42 ± 0.01 (-2.3%)
1.89 ± 0.05 (-6.9%)
0.46 ± 0.01 (-2.1%)
+ GUT-O
43.00 ± 1.67
−14.03 ± 0.07
0.39 ± 0.01 (-9.3%)
1.82 ± 0.05 (-10.3%)
0.43 ± 0.01 (-8.5%)
Qwen3-4B
53.80 ± 1.71
−14.14 ± 0.08
0.37 ± 0.01
1.76 ± 0.05
0.40 ± 0.01
+ GRPO
61.00 ± 1.68
−14.11 ± 0.09
0.38 ± 0.01 (+2.7%)
1.78 ± 0.04 (+1.1%)
0.40 ± 0.01 (+0.0%)
+ GUT-O
57.00 ± 1.71
−14.25 ± 0.07
0.34 ± 0.00 (-8.1%)
1.70 ± 0.04 (-3.4%)
0.32 ± 0.01 (-20.0%)
Qwen3-8B
64.20 ± 1.68
−15.14 ± 0.09
0.33 ± 0.01
1.46 ± 0.04
0.24 ± 0.01
+ GRPO
66.60 ± 1.64
−14.97 ± 0.09
0.33 ± 0.01 (+0.0%)
1.43 ± 0.04 (-2.1%)
0.25 ± 0.01 (+4.2%)
+ GUT-O
64.80 ± 1.72
−15.78 ± 0.08
0.31 ± 0.01 (-6.1%)
1.38 ± 0.04 (-5.5%)
0.23 ± 0.01 (-4.2%)
Table 3: Comparisons of UO performance for 4 Qwen3 LLMs on the MATH-500 dataset, where ‘+’ in column “Models” indicates the applied fine-tuning method, and values in parentheses denote the percentage change in mean uncertainty. Table 3 shows the comparisons of UO performance for 4 Qwen3 LLMs on the MATH-500 dataset, where ‘+’ indicates the applied fine-tuning method, and values in parentheses indicate the percentage change in mean uncertainty. There are two key observations. First, after employing GUT-O, all three GUT-Q-derived mean uncertainties decrease while the accuracy improves. This observation validates the effectiveness of GUT-O. Second, it is observed that mean uncertainties are typically comparable or even increase after applying GRPO, which suggests that the UO performance of GUT-O primarily benefits from the MTLP optimization target rather than task-specific fine-tuning. Similar observations hold for all 4 evaluated LLMs and 5 datasets, as shown in Table 14 in Appendix C.2. These findings
13
demonstrate that GUT-O separately reduces reasoning uncertainty and improves accuracy by an average of 13.95% and 1.93% across 4 LLMs and 5 datasets, which answers Q2.
6. Conclusions
In this paper, we investigated the reasoning uncertainty of LLMs by using DAGs to characterize the potential branches of each reasoning step. Built upon this recognition, we proposed the GUT comprising GUT-Q and GUT-O for quantifying and reducing reasoning uncertainty, respectively. The GUT-Q quantifies reasoning uncertainty by the reasoning space complexity, which is approximated by the complexity of the DAG. Defining the negative uncertainty proxy as the reward in the GRPO method, the GUT-O further reduces reasoning uncertainty. Empirical results showed that GUT-Q-UP surpasses the best UQ contenders by an average of 11.79% in PRR, 13.33% in AUROC, and 9.66% in AUPRC. GUT-O reduces reasoning uncertainty and improves accuracy by an average of 13.95% and 1.93%, respectively.
Acknowledgments
The research was supported by the Natural Science Foundation of China (62406138).
14
Appendix
This appendix provides the supplementary materials for our work, constructed according to the corresponding sections therein. Before that, we review the origin of reasoning uncertainty in LLMs, which has been illustrated in Figure 1 and introduced in Section 3. Given a problem, the stochasticity of the LLM system originates from the token-level distribution. The autoregressively stochastic temperature sampling over this distribution yields variability in the output reasoning chains, which ultimately gives rise to reasoning uncertainty. This work focuses on quantifying and optimizing reasoning uncertainty.
A. Additional Implementation Details of GUT
A.1. Prompt Templates This subsection details the prompt templates used for the CoT sampling in the GUT-Q module and UQ contenders. Due to space constraints, we only list the templates of 0-shot and 1-shot here. 0-shot Prompt Answer the following problem. Break down your reasoning process into small steps. Each step should represent a single, minimal reasoning action, and each step must logically follow the previous one. Use the following format for each step: Step i: [Your reasoning process in one cohesive response] After completing all the steps, conclude with: Final Answer: \boxed[Your final answer here without the unit or any additional text] #### Stop generation immediately after outputting the Final Answer. Ensure that your response strictly follows the format to maintain clarity and consistency. Problem: [INPUT_PROBLEM] Reasoning:
15
1-shot Prompt Answer the following problem. Break down your reasoning process into small steps. Each step should represent a single, minimal reasoning action, and each step must logically follow the previous one. Use the following format for each step: Step i: [Your reasoning process in one cohesive response] After completing all the steps, conclude with: Final Answer: \boxed{[Your final answer here without the unit or any additional text]} #### Stop generation immediately after outputting the Final Answer. The following is an example: Problem: Let M be the midpoint of AB in regular tetrahedron ABCD. pq = cos(∠CM D) is irreducible fraction, what is the value of p + q? Reasoning: Step 1: Without loss of generality, let the edge-length of ABCD be 2. It follows that √ M C = M D = 3. Step 2: Let O be the center of △ABD, so CO ⊥ M OD. Note that M O = 13 M D =
√
3 3 .
O 1 Step 3: In right △CM O, we have cos(∠CM D) = M MC = 3 .
Step 4: So the answer is 1 + 3 = 4. Final Answer: \boxed{4} ####
Problem: [INPUT_PROBLEM] Reasoning: For the GSM8K and MATH-500 datasets, we use 7 instances from the training sets of GSM8K and MATH [18], respectively, to construct the few-shot prompts. For the AMC2022-2024 dataset, we use 7 instances from the AMC2022-20231 split to construct the few-shot prompts and remove these instances from the constructed AMC2022-2024 dataset. We append #### after “Final Answer” to serve as a stopping string for efficient sampling. By default, we follow [54] and use a 3-shot prompt to extract the reasoning steps along with the final answer. We also conduct sensitivity analyses on the number of shots in Subsection 5.1 and Appendix D.2.
A.2. Details of GUT-Q Algorithm This subsection first lists several algorithms regarding graph complexity estimation proposed in Subsection 3.3, and then provides the pseudocode of the graph construction procedure in GUT-Q. 1 https://huggingface.co/datasets/AI-MO/aimo-validation-amc
16
Algorithm 3 GUT-Q-H Input: DAG G = (V, E), node uncertainty {U (v)}v∈V Output: Uncertainty-weighted height Hu Procedures: 1: L ← TopologicalSort(G) 2: R[v] ← 0 for all v ∈ V {Initialize topological level} 3: for u ∈ L do
R[u] ← max({R[p] + 1 | (p, u) ∈ E} ∪ {0})
4:
5: end for 6: Vh ← GetHeightNodes(G, R) {Get height-path nodes} 7: Hu ←
P
v∈Vh U (v)
Algorithm 3 lists the GUT-Q-H graph complexity method introduced in Subsection 3.3, where its output Hu serves as a measure of graph complexity and is further used to quantify reasoning uncertainty. Algorithm 4 GUT-Q-UP-Weights Input: DAG G = (V, E), node uncertainty {U (v)}v∈V , bias ρ ∈ R+ , activation function ϕ Output: Auxiliary node uncertainty U (sa ) Procedures: 1: Construct augmented graph G′ = (V ′ , E ′ ) by connecting all leaf nodes of G to a new auxiliary node
sa 2: Construct FNN corresponding to G′ with ϕ 3: Set weights wuv ← U (v) for all (u, v) ∈ E ′ 4: Set biases bv ← ρ for all v ∈ V , and bsa ← 0 5: U (sa ) ← output of sa from forward propagation
Algorithm 4 is a variant of Algorithm 2. The key idea is to initialize the weights of the FNN with the node uncertainty and the biases with a hyperparameter ρ ∈ R+ . This variant is dual to Algorithm 2, which initializes the biases with the node uncertainty and the weights with the hyperparameter ω ∈ R+ . We note that the UQ performance of Algorithms 2 and 4 is comparable across all concerned LLM scales, datasets, and UQ evaluation metrics, as shown in Table 4. Algorithm 5 summarizes the DAG construction procedure in Subsection 3.1.
A.3. Details of GUT-O Formulation This subsection formally introduces GUT-O. We start with the calculation of the MTLP, which is a special case of the step-level uncertainty calculation in Subsection 3.2. Specifically, we compute the 17
GSM8K
MATH-500
AMC2022-2024
Scale UQ
GUT-Q-UP
PRR
AUROC
AUPRC
PRR
72.01±4.34
83.72±1.55
97.01±0.50 53.28±4.34
AUROC
AUPRC
75.93±1.90 81.59±1.87
PRR 72.85±4.89
AUROC
AUPRC
81.83±3.02 84.53±2.88
8B GUT-Q-UP-W 73.91±4.19 84.42±1.60 97.39±0.46
52.77±4.57 76.64±1.76
79.34±2.11 74.68±6.32 87.35±2.54
GUT-Q-UP
61.93±3.59
79.45±2.07 80.82±4.24 89.48±2.19 84.60±3.39
81.68±1.83
86.61±1.02
96.93±0.36
80.13±1.57
82.27±4.55
4B GUT-Q-UP-W 81.76±1.79 86.63±1.00 96.94±0.35 62.13±3.60 80.23±1.58 79.56±2.15 85.48±0.86 92.83±0.67
79.07±4.50
88.72±2.19
83.24±3.72
74.08±2.45 68.00±6.93 82.90±3.72 68.00±6.30
GUT-Q-UP
77.50±1.83
GUT-Q-UP-W
77.29±1.92 85.49±0.86
92.76±0.70 63.95±3.36 81.13±1.51 74.39±2.44
66.56±7.22
81.68±3.90
66.99±6.43
GUT-Q-UP
52.02±2.51
57.57±2.06 52.04±4.58 76.50±2.31 48.71±4.35
14.75±8.38 67.52±nan
15.48±4.79
63.39±3.36
80.68±1.50
1.7B 75.31±1.20
0.6B GUT-Q-UP-W 52.17±2.51 75.45±1.20 57.64±2.06
50.37±4.44
75.05±2.30
48.14±4.33 14.90±8.74
67.11±nan 15.59±5.05
Table 4: Comparisons of UQ performance between the GUT-Q implemented with Algorithms 2 (GUTQ-UP) or 4 (GUT-Q-UP-W). “Avg Log Prob.” for each token in c, and set w = 1 and d = 100 to perform the Top-d% aggregation over c to yield U (x, c). Before listing the optimization procedures, we introduce some useful notations. Let clip(a, ϵ) = max(1 − ϵ, min(a, 1 + ϵ)) for ϵ ∈ (0, 1) and a ∈ R. We denote the current, old, and reference policy models as πθ , πθold , and πref , respectively. With the reward defined as the negative uncertainty proxy ri = −U (x, ci ) and M ∈ N+ , the GUT-O optimizes πθ by maximizing i
|c | M 1 X 1 X min lji (θ), clip lji (θ), ϵ Ai − βKL [πθ ||πref ] J (θ) = Ex∼X ,{ci }i∈[M ] ∼πθold (·|x) i M i=1 |c | j=1
,
where h i lji (θ) = πθ (tij |hij ) / πθold (tij |hij ), Ai = ri − mean ri i∈[M ] / std ri i∈[M ] , and KL [πθ ||πref ] = πref (tij | hi<j ) / πθ (tij | hi<j ) − log πref (tij | hi<j ) / πθ (tij | hi<j ) − 1 . Within the optimization target J (θ), the first term represents the clipped surrogate advantage. By leveraging the normalized advantage Ai , it encourages the model to increase the probability of generating reasoning paths that yield higher rewards, i.e., lower MTLP. Meanwhile, the clipping mechanism ensures stable training by restricting overly large policy updates. The second term introduces a Kullback-Leibler (KL) divergence penalty, scaled by the coefficient β ∈ R. This term acts as a regularizer to prevent the active policy πθ from deviating too drastically from the reference model πref , thereby mitigating reward hacking and preserving the fundamental language capabilities of the LLM.
18
Algorithm 5 DAG Construction Input: problem x, sampling times K, temperature T Output: DAG G = (V, E) Procedures: 1: Initialize G = (V, E) with root node x 2: for i ∈ [K] do 3:
Sample a reasoning chain si at temperature T
4:
Extract reasoning steps {sij }j∈[ni ] from si
5:
Calculate step-level uncertainty {U (sij )}j∈[ni ]
6:
Initialize U C(sij ) ← {U (sij )} for all j ∈ [ni ]
7:
V ← V ∪ {sij }j∈[ni ] , E ← E ∪ {(x, si1 )}
8:
E ← E ∪ {(sij , sij+1 )}j∈[ni −1]
9: end for 10: Initialize processed node set Vp ← ∅ {Node merging} 11: for i ∈ [K] do 12:
Initialize node set Vm ← Vp for merging {AOV-based traversal}
13:
Vp ← Vp ∪ {sij }j∈[ni ]
14:
for j ∈ [ni ] do
15: 16:
for each node v ∈ Vm do if NLI predicts bi-entailment for v and sij then
17:
U C(v) ← U C(v) ∪ U C(sij )
18:
Get the parent p of sij where (p, sij ) ∈ E
19:
E ← E ∪ {(p, v), (v, c) | (sij , c) ∈ E}
20:
Vp ← Vp \ {sij } , Vm ← Vm \ {v}
21:
Vm ← Vm \ {u} for all ancestor nodes u of v
22:
break
23:
end if
24:
end for
25:
end for
26: end for 27: U (v) ← Average(U C(v)) for all v ∈ Vp 28: Update G to retain only nodes in Vp ∪ {x}
19
B. Additional Experimental Details on GUT-Q
This appendix provides additional experimental results and implementation details on the proposed GUT-Q module.
B.1. Details on Datasets In this work, we conducted experiments on mathematical reasoning benchmarks across three difficulty levels, ordered from easy to hard as follows. GSM8K contains 8500 grade school math word problems. We use the test set of GSM8K, which contains 1319 problems. MATH-500 [21] consists of 500 high school math problems. AMC2022-2024 comprises 128 high school math competition problems, constructed by aggregating the AMC2022-20232 and AMC20243 . All problems are fill-in-the-blank, where the final answer is typically a real number or a real vector. The difficulty level is characterized by the accuracy. In our implementation, we use the Math-Verify4 package to verify the correctness of the extracted final answer. Table 5 shows the accuracy of the evaluated LLMs on the concerned datasets. It is observed that the accuracy decreases sequentially across GSM8K, MATH-500, and AMC2022-2024 for all LLM scales, indicating that the difficulty level increases sequentially across GSM8K, MATH-500, and AMC2022-2024. Mathematical Reasoning
First-Order Logic Reasoning
Reasoning QA
Scale GSM8K
MATH-500
AMC2022-2024
FOLIO
MMLU-Pro
8B
89.61
64.20
49.59
72.41
69.58
4B
84.08
53.80
38.02
68.47
60.83
1.7B
71.11
42.20
22.31
43.84
43.33
0.6B
28.89
18.00
9.09
44.83
29.17
Table 5: Accuracy of the evaluated LLMs on the concerned datasets.
We note that the presented accuracy may differ from that of [48], since (1) we employ a different prompt template to extract the reasoning steps and final answers for UQ, and (2) we use a smaller token generation budget, as detailed in Appendix B.2, due to the constraints of computational resources. However, this work mainly focuses on quantifying and optimizing the reasoning uncertainty, instead of improving the accuracy. Therefore, the accuracy differences between ours and that of [48] are acceptable. 2 https://huggingface.co/datasets/AI-MO/aimo-validation-amc 3 https://huggingface.co/datasets/rawsh/2024_AMC12 4 https://github.com/huggingface/Math-Verify
20
We also note that the low accuracy of Qwen3-0.6B on the AMC2022-2024 dataset often leads to single-class bootstrapping where all samples are incorrect, which results in the nan values of std in Tables 4 and 12 and missing std bars in Figure 5d since the AUROC is undefined for binary classification tasks containing only a single class. Moreover, we conducted experiments on first-order logic reasoning tasks such as FOLIO [16] and challenging long-form QA tasks that may require reasoning like MMLU-Pro [45], to examine the generality of the proposed GUT. The instances of these two datasets are multiple-choice questions, where the final answer of FOLIO is within {True, False, Uncertain}, and that of MMLU-Pro is within {A, B, C, D}.
B.2. Configurations of Hyperparameters This subsection details configurations of hyperparameters in the GUT-Q module.
Configuration of the Step-level Uncertainty Calculation. Following Fu et al. [14], we set the hyperparameters for the “step-level uncertainty calculation” stage of GUT-Q as shown in Table 6. Hyperparameter
Setting
Token-level UQ
Avg Log Prob
Selection Strategy
Top-3%
Window Size w
6
Weight ω in Algorithm 2
0.2
Table 6: Configuration of hyperparameters in step-level uncertainty calculation.
Configuration of Sampling Parameters. Following prior work [26, 12, 32], we use different sampling setups for evaluating correctness and quantifying uncertainty.
• We follow the official Qwen3 recommendation5 using T = 0.7, Top-k = 20, and Top-p = 0.8 to obtain the reasoning chain which is compared to the reference answer. This sample determines whether the problem is correctly answered. • We follow LM-Polygraph6 [40] and set T = 1.0 and K = 10 to obtain the candidate reasoning chains for ensuring a fair comparison with the UQ contenders. We also follow the official Qwen3 recommendation to set Top-k = 20 and Top-p = 0.8.
21
Figure 3: Complete generation rate. We conducted sensitivity analyses to the max_new_tokens hyperparameter for the concerned datasets. The max_new_tokens controls the maximum number of tokens that the LLM can generate. Hence, a larger max_new_tokens indicates higher computational cost. However, a lower max_new_tokens may lead to incomplete generations, such as missing the “Final Answer” that is required in our prompt template provided in Appendix A.1. Therefore, there is a trade-off between the max_new_tokens and the complete generation rate. Figure 3 shows the complete generation rate versus the max_new_tokens. We set max_new_tokens = 256 for FOLIO, max_new_tokens = 512 for GSM8K and MMLU-Pro, max_new_tokens = 1536 for MATH-500, and max_new_tokens = 2048 for AMC2022-2024 to achieve the concerned trade-off across all LLM scales.
B.3. UQ Contenders Table 7 lists the full names and corresponding abbreviations (Abbr.) of the 45 UQ contenders. We implemented CoT-UQ7 [52] and TopoUQ8 [8] using their official repositories, respectively, and the remaining contenders via LM-Polygraph9 [40].
5 https://huggingface.co/Qwen/Qwen3-4B 6 https://github.com/IINemo/lm-polygraph 7 https://github.com/ZBox1005/CoT-UQ 8 https://github.com/LongchaoDa/LLM-Topology 9 https://github.com/IINemo/lm-polygraph
22
Abbr.
Full Name
Type
Category
Paper
MSP
Maximum Sequence Probability
Fadeeva et al. [11]
Ppl
Perplexity
Fomicheva et al. [13]
MTE
Mean Token Entropy
Fomicheva et al. [13]
MTLP
Mean Token Log Probability
Manakul et al. [26]
MPMI
Mean Pointwise Mutual Information
Takayama and Arase [38]
CPMI
Conditional Pointwise Mutual Information
van der Poel et al. [39]
RD
Rényi Divergence
Darrin et al. [9] Information-based
FRD
Fisher-Rao Distance
Darrin et al. [9]
AS
Attention Score
Sriramanan et al. [37]
CSL
Contextualized Sequence Likelihood
Lin et al. [22]
MCSE
Monte Carlo Sequence Entropy
Malinin and Gales [25]
MCNSE
Monte Carlo Normalized Sequence Entropy
Malinin and Gales [25]
RAUQ
Recurrent Attention-based UQ
Vazhentsev et al. [42]
RAUQ-E
Recurrent Attention-based UQ-Entropy
White-box
Vazhentsev et al. [42]
SE
Semantic Entropy
SAR
Shifting Attention to more Relevant
Farquhar et al. [12] Duan et al. [10]
TSAR
TokenSAR
Duan et al. [10]
SSAR
SequenceSAR
SD
Semantic Density
ES
Eigen Score
Chen et al. [4]
CoCoA-MSP
CoCoA-Mean Sequence Probability
Vashurin et al. [41]
CoCoA-Ppl
CoCoA-Perplexity
Vashurin et al. [41]
CoCoA-MTE
CoCoA-Mean Token Entropy
Vashurin et al. [41]
P(True)
P(True)
CoT-UQ-ME
CoT enhanced UQ-probas MEan
CoT-UQ-MI
CoT enhanced UQ-probas MIn
CoT-UQ-SAR
CoT enhanced UQ-TokenSAR
Zhang and Zhang [52]
NS
Number of Sets
Lin et al. [23]
Duan et al. [10] Diversity-based
Qiu and Miikkulainen [32]
Reflexive-based
Kadavath et al. [20]
Reasoning-enhanced
Zhang and Zhang [52]
Zhang and Zhang [52]
Eig-E
Sum of Eigenvalues-NLI Score Entail.
Lin et al. [23]
Eig-C
Sum of Eigenvalues-NLI Score Contra.
Lin et al. [23]
Eig-J
Sum of Eigenvalues-Jaccard Score
Lin et al. [23]
Deg-E
Degree Matrix-NLI Score Entail.
Lin et al. [23]
Deg-C
Degree Matrix-NLI Score Contra.
Lin et al. [23]
Deg-J
Degree Matrix-Jaccard Score
Lin et al. [23]
Ecc-E
Eccentricity-NLI Score Entail.
Ecc-C
Eccentricity-NLI Score Contra.
Lin et al. [23] Diversity-based Black-box
Lin et al. [23]
Ecc-J
Eccentricity-Jaccard Score
Lin et al. [23]
LS-R1
Lexical Similarity-Rouge 1
Farquhar et al. [12]
LS-R2
Lexical Similarity-Rouge 2
Farquhar et al. [12]
LS-RL
Lexical Similarity-Rouge L
Farquhar et al. [12]
LS-B
Lexical Similarity-BLEU
Farquhar et al. [12]
KLE
Kernel Language Entropy
Nikitin et al. [30]
LUQ
Long-text Uncertainty Quantification
Zhang et al. [53]
CoT-UQ-SP
CoT enhanced UQ-Self Probing
Reasoning-enhanced
Zhang and Zhang [52]
Topo-UQ
Topology-based UQ
Graph-based
Da et al. [8]
Table 7: Overview of the investigated UQ contenders. 23
B.4. Runtime Complexity Analysis of GUT-Q and Its Contenders This subsection provides a runtime complexity analysis of GUT-Q and its contenders. Specifically, we select eight representative methods, namely Ppl, MCSE, SE, P(True), CoT-UQ-ME, Eig-E, CoT-UQ-SP, and Topo-UQ, to cover all investigated categories. The runtime of the considered UQ methods primarily consists of three parts arranged in decreasing order of computational cost, namely LLM inference, auxiliary model execution, and other specific computations such as calculating token-level statistics and deriving the eigenvalues of the similarity matrix. Modern LLMs like Qwen3 typically operate at the billion-parameter scale, whereas auxiliary models usually have millions of parameters. Thus, LLM inference is significantly longer than that of the auxiliary model, which makes the overall runtime of a UQ method primarily dominated by the number of LLM inferences. Number of LLM Category
Other Computation Inferences
Informationbased Reflexive-
Auxiliary Model Execution
UQ Model
Scale
Number of Execution
Ppl
1
-
-
-
Token-level statistics computation
MCSE
K
-
-
-
Token-level statistics computation
P(True)
2
-
-
-
-
CoT-UQ-ME
2
-
-
-
Token-level statistics computation
CoT-UQ-SP
3
-
-
-
-
SE
K
DeBERTa
0.35B
based Reasoningenhanced
K (Best case)
Diversitybased
Token-level statistics computation
K(K − 1)/2 (Worst case) Eig-E
K
Topo-UQ
L(n + 2)
GUT-Q
K
0.35B
K(K − 1)/2
Eigen value computation
BERT-Base 0.11B
L(|V | + |E|)
Graph edit distance computation
KS (Best case)
Token-level statistics computation
DeBERTa
Graphbased
DeBERTa
0.35B
KS(K − 1)(S − 1)/4 (Worst case) Graph complexity estimation
Table 8: Runtime complexity comparison of GUT-Q and its contenders. Table 8 shows the runtime complexity comparison of GUT-Q and its contenders, where K ∈ N+ is the number of CoT samples, S ∈ N+ denotes the average number of reasoning steps in a CoT chain, L ∈ N+ marks the per-specified number of constructed graphs, n ∈ N+ indicates the number of subproblems decomposed by prompting the LLM, |V | ∈ N+ is the number of vertices in the graph, and |E| ∈ N+ is the number of edges in the graph. Because SE and our proposed GUT-Q rely on chain-level or step-level merging guided by an auxiliary model, their best-case scenario occurs when all chains or steps belong to the same cluster, while the worst-case scenario arises when each forms a distinct cluster. There are two key observations. First, compared to representative diversity-based UQ methods such as SE and Eig-E, our proposed GUT-Q shares the same number of LLM prompts while requiring a slightly larger number of auxiliary model executions. Second, compared with classical graph-based methods like Topo-UQ, our proposed GUT-Q exhibits a comparable number of LLM prompts and auxiliary model executions. The 24
above two observations suggest that our proposed GUT-Q runtime complexity is comparable to the investigated diversity-based and graph-based UQ methods, which typically achieve relatively better UQ performance in selective generation than the concerned information-based methods and reflexive-based methods, as shown in Section 5.
Figure 4: Average wall-clock time comparison of GUT-Q and its contenders on the MATH-500 dataset using Qwen3-4B. Figure 4 further visualizes the average wall-clock time comparison of GUT-Q and its contenders on the MATH-500 dataset using Qwen3-4B, where we set K = 10 and L = 5 to make LLM inference cost comparable. It is observed that GUT-Q exhibits a wall-clock time comparable to existing state-of-the-art methods such as SE, Eig-E, and Topo-UQ, which corroborates the aforementioned runtime analysis and corresponding conclusions. We can also observe that the sampling time of GUT-Q is only around 2.5 times that of the Ppl method, far less than K = 10 times. This is because one can perform batch-parallel LLM prompting in implementation using packages like Transformers 10 or vllm 11 , which makes our proposed GUT-Q more efficient in practice.
B.5. UQ Performance Evaluations Tables 9, 10, 11, and 12 show the UQ performance comparisons across 5 datasets for Qwen3-8B, Qwen3-4B, Qwen3-1.7B, and Qwen3-0.6B, respectively. It is observed that the GUT-Q implemented with the UP algorithm generally outperforms all contenders across three evaluation metrics and datasets spanning three difficulty levels, thus validating the proposed GUT-Q.
10 https://huggingface.co/docs/transformers/en/index 11 https://vllm.ai/
25
26
11.71±0.94
11.71±0.94
-3.45±0.98
-3.01±0.95
MCSE
MCNSE
RAUQ
RAUQ-E
19.04±2.34
20.71±2.33
13.97±2.03
2.83±2.22
94.10±0.64
87.36±1.05 20.17±5.72
-3.49±2.40
90.34±0.87 -22.41±2.36
88.89±0.92 -22.63±2.38
90.60±0.90 -11.40±2.27
87.52±1.01
87.04±1.03
91.68±0.74 -24.84±2.52
87.10±0.82
83.69±1.14
89.84±0.73 -16.42±2.54
89.08±0.90 -22.07±2.49
91.49±0.80 -24.61±2.55
89.54±0.68 -13.21±2.51
89.54±0.68 -13.21±2.51
28.51±1.68
36.75±0.69
43.02±0.63
54.66±0.50
38.95±0.69
37.43±0.64
42.66±0.60
42.66±0.61
38.15±0.67
-1.41±0.95
48.89±0.59
Deg-J
Ecc-E
Ecc-C
Ecc-J
LS-R1
LS-R2
LS-RL
LS-B
KLE
LUQ
58.26±4.10
35.32±6.10
GUT-Q-W
GUT-Q-H
41.93±5.38 49.66±4.58
56.08±4.00 23.22±3.36
52.55±2.16
34.95±2.03
50.93±2.20
59.32±2.21
49.93±0.54
59.51±2.08
59.33±2.19
59.23±2.14
47.22±2.24
54.77±1.97
55.56±2.11
53.85±2.01
34.74±2.17
54.32±2.18
49.52±2.23
41.64±2.07
49.89±2.23
37.09±2.16
34.72±2.10
37.47±2.09
43.12±2.12
31.98±2.01
44.06±2.00
73.31±1.92
70.95±2.01
25.36±5.09
44.03±4.24
42.75±4.16
52.51±3.65
54.80±3.68
40.77±9.29
54.56±3.94
56.32±3.88
2.01±4.67
80.34±1.96
77.59±2.16
67.11±2.32
63.09±2.12
61.40±2.23
64.39±2.06
57.03±2.13
55.10±2.16
56.62±2.21
48.93±3.32 -18.31±3.38 19.79±3.42
12.20±3.33
12.36±3.32
38.35±3.01
2.90±3.27
23.30±3.17
10.98±3.33
72.77±2.51
72.77±2.42 0.12±3.46
0.12±3.46
76.38±3.47 -10.78±3.49
76.74±3.13
75.19±2.79
75.10±2.88
69.64±3.34
78.77±3.02
78.26±3.15
70.47±3.23
74.18±2.77 -18.28±3.51
29.89±2.94
42.82±2.32
50.74±2.31
4.61±3.17
55.86±2.07
49.40±2.37
5.40±3.31
28.11±7.97
29.37±8.39
28.83±7.86
-7.66±3.24
-6.91±3.05
-9.20±3.12
-8.46±3.10
27.30±3.10
53.61±4.52
57.82±5.15
49.04±4.29
36.78±3.49
36.07±3.59
37.67±3.60
45.89±4.92
37.52±3.44
60.33±5.05
60.79±5.34
39.01±3.93
14.77±8.61
58.54±2.02
-6.33±3.52
-3.30±3.44
10.99±3.40
0.98±3.46
-7.38±3.38
13.66±3.13
58.75±1.97
45.09±2.43
4.40±3.20
57.82±5.04 60.15±1.95
59.07±5.25
39.05±3.76
60.34±5.12
60.82±5.33
50.41±3.56
70.37±5.04
71.85±4.68
70.82±4.98
36.64±4.56
31.76±3.35
31.69±3.35
30.84±3.44
31.25±3.80
21.46±2.99
22.29±3.04
48.35±3.49 -14.84±3.45
36.89±3.71
58.12±4.65
52.58±7.46 57.58±7.01
82.51±2.45
78.58±2.84
78.11±3.06
78.58±3.10
78.85±3.17
62.49±3.48
67.59±3.51
31.18±3.21
74.30±3.02
71.27±3.09
59.00±3.52
51.22±1.88
27.75±2.99
51.86±3.56
50.32±3.58
42.68±3.75
48.65±3.68
54.39±3.42
43.96±3.72
28.30±2.89
33.59±3.23
49.08±3.64
27.26±3.03
30.87±3.20
49.63±3.65
29.34±3.14
82.34±2.68
44.72±2.63
13.84±3.28
60.68±2.16
13.99±3.37
2.61±3.40
28.50±9.66
25.89±3.18
55.22±2.28
7.95±3.39
16.97±3.25
9.78±3.38
3.27±3.40
-0.96±3.43
65.98±2.09
87.49±2.31
86.15±2.41
77.96±3.09
45.43±7.78
55.09±8.05
35.93±6.56
77.66±2.25 -8.58±10.88
61.33±3.19
74.74±2.95 -10.74±3.51
73.54±3.29
67.62±3.36
72.09±3.37
77.22±3.18 -22.40±3.50
68.75±3.45
61.97±3.46
64.24±3.26
73.38±3.18
61.89±3.35 67.68±2.04
62.58±3.31
74.04±3.39
63.04±3.27
62.61±3.43
71.84±2.57
53.39±3.12
61.90±3.07
45.12±3.04
56.66±3.10
58.88±2.86
55.58±3.12
29.55±2.90
33.32±3.00
AUPRC
80.54±3.01
63.93±3.40
71.70±3.30
74.16±3.22
69.77±3.36
53.71±2.86
57.64±3.32
69.23±3.21
57.35±2.58
54.62±3.09
65.61±2.77
78.54±3.18
75.38±3.09
63.33±2.51
63.33±2.31
78.41±2.72
63.19±3.24
64.90±2.72
65.14±2.70
61.38±3.25
76.54±2.91
72.21±3.44
55.69±3.18
69.52±2.75
69.52±2.79
72.22±2.89
75.51±2.57
62.43±3.09
47.29±3.23
22.48±2.36
49.64±2.93
42.23±3.18
44.47±3.21
45.75±3.07
55.66±3.19
37.38±2.95
29.95±2.64
45.69±3.10
42.28±3.09
21.15±2.20
33.12±2.76
43.31±3.12
26.04±2.55
41.99±3.07
47.31±1.29
61.61±3.02
82.36±2.83
85.72±2.57
75.69±2.91
71.43±2.77
48.84±2.81
63.72±3.16
58.05±3.37
60.27±3.14
60.01±3.29
66.43±3.24
56.05±3.13
53.09±3.04
59.79±3.29
58.31±3.14
47.79±2.75
53.86±3.10
58.55±3.40
50.88±3.06
57.81±3.44
62.62±2.47
80.35±2.85
37.25±9.34 66.43±3.07 82.62±2.69
29.35±9.56
22.81±3.31
6.64±3.38
3.83±3.39
6.66±3.28
54.44±2.96
44.09±3.13
37.70±2.62
34.83±2.94
54.35±2.80
62.15±2.87
60.05±3.12
50.00±0.00
50.00±0.00
61.88±2.93
45.55±3.08
41.66±2.58
41.68±2.61
40.38±2.88
58.16±3.09
53.58±2.92
40.37±3.17
59.41±2.58
59.41±2.66
AUROC
MMLU-Pro
Table 9: UQ performance comparisons for Qwen3-8B, where bold and underline denote the group best and second-best results.
81.83±3.02 84.53±2.88 62.41±7.84 79.07±2.75 88.85±2.49 69.80±6.26 83.56±2.33 89.86±2.11
79.01±3.51 83.98±3.43
57.01±3.51 44.24±2.12
40.85±3.26
30.18±3.18
71.67±3.19 -16.53±3.55
69.36±3.02
66.46±3.46
69.32±2.73 -13.28±3.59
58.93±3.53 82.57±2.66
57.22±3.53
55.60±3.37
57.77±3.34
58.78±3.65
59.02±3.37
35.03±3.59
38.87±3.51
42.37±3.56
44.74±3.22
37.51±3.38
41.28±1.91
78.54±2.99 -12.55±3.52
74.23±3.59
61.79±5.82
PRR
31.34±3.17 -19.91±3.28 60.73±3.33
68.75±4.14 71.90±5.23 20.64±10.41
49.35±4.36
58.63±4.23
47.24±4.22
23.91±3.24
21.80±3.28
26.78±3.60
40.55±4.03
AUPRC 74.18±2.78 -18.28±3.51
79.77±3.11 -13.45±3.51
49.66±0.26
49.66±0.29
55.64±3.76
55.42±3.76
48.81±3.07
48.74±3.00
41.47±3.19
58.87±3.41
58.87±3.37
44.63±3.37
51.63±3.03
51.63±3.05
AUROC
FOLIO
60.50±3.30
9.21±3.27
9.21±3.27
-9.70±3.41
31.79±3.38 -21.88±3.34
51.26±3.54
51.26±3.66
33.71±3.75
72.80±5.44 83.52±2.97
42.73±9.96
6.43±11.47
-8.33±4.90
17.05±5.01
60.86±3.86
60.14±4.04
52.58±4.23
29.27±4.67
23.72±3.28
53.10±1.91 67.64±3.60 61.65±2.30
57.87±4.13
61.94±4.05
28.31±3.67
56.04±4.24
62.87±4.11
28.88±3.87
62.62±3.96
64.13±4.05
50.00±0.00
70.81±3.80
65.87±4.16
71.03±3.98
33.00±4.10
22.31±3.58
22.18±3.39
18.87±3.53
19.37±3.40
63.00±2.23 -13.16±5.07
67.62±2.27 -26.85±5.13
54.78±2.06
59.14±2.33
66.96±2.29 -32.00±4.91
55.09±2.08
64.64±2.36 -32.15±4.97
69.79±2.44 -36.56±5.00
65.28±1.71 -10.00±4.93
71.18±2.27 36.95±11.07
70.77±2.23
70.64±2.24 38.27±10.98
58.81±2.29
68.43±2.01
69.26±1.93
67.29±2.12
49.63±1.97
29.67±2.77
-9.25±3.11
32.37±3.66 -13.92±3.52
68.17±4.86
64.39±3.64
1.53±3.27 15.27±3.10
48.99±3.34 -18.25±3.37
65.59±5.09
47.89±5.10
62.37±4.79
62.02±1.65
57.32±2.18
20.39±3.38
22.23±3.31
51.67±0.92
51.67±0.93
25.54±3.89
23.58±3.73
25.60±2.91
25.63±2.83
73.58±3.40
PRR 1.53±3.27
29.89±3.33 -10.29±3.16
34.88±3.33
67.61±4.01
57.51±4.10
28.64±5.09 62.44±1.99
-9.07±6.34
23.47±2.13
-0.59±2.36
27.65±2.25
30.54±2.28
25.04±2.31
16.16±2.35
28.84±2.36
39.10±4.46
70.33±1.84 -15.79±5.04
68.50±2.05
35.77±4.49
66.05±1.64 -16.80±4.82 67.24±2.13
72.01±4.34 83.72±1.55 97.01±0.50 53.28±4.34 75.93±1.90 81.59±1.87 72.85±4.89
93.21±0.74
95.56±0.52
93.85±0.59
74.22±1.88
91.88±0.80
54.91±2.22
13.04±6.73
45.31±4.24 66.04±1.82
83.66±1.13
89.09±0.86
84.93±1.10
84.26±1.15
84.41±1.17
85.41±1.09
84.14±1.16
15.97±2.21
31.34±4.60
37.50±4.32
53.11±4.73
54.07±4.75
66.05±1.63 -16.80±4.82
67.51±2.35
58.43±2.19
65.54±1.67
65.38±1.74
49.81±4.19
15.07±3.25
17.60±2.96
74.62±2.76
AUPRC 67.62±4.05
59.65±2.31 -73.15±3.85 75.87±3.46
34.43±1.98
83.69±1.16
-0.05±5.14
61.95±3.79
60.21±4.34
58.95±2.26 -62.38±4.13
69.99±2.01
66.44±2.12
54.67±2.11
70.09±1.77 -61.57±4.10
74.67±2.88
AUROC
61.42±1.90 73.04±1.84 -17.97±4.98 44.60±2.10
84.96±1.15 33.59±2.19 83.76±1.17 -17.41±2.54
PRR
24.54±2.75
36.69±2.12
-2.70±2.38
32.25±2.21
AUPRC 70.09±1.76 -62.22±4.03
AMC2022-2024
51.48±4.77
39.72±1.52
39.85±2.13
58.58±1.96
55.64±1.94
55.37±2.07
51.85±0.58
51.85±0.63
56.14±1.99
43.59±2.05
41.06±1.53
41.02±1.48
30.80±2.13
83.21±1.12
84.08±1.14
84.91±1.14
83.53±1.15 -13.53±2.51
Topo-UQ
64.87±2.02
13.16±2.41
84.13±1.16 -40.48±2.57
CoT-UQ-SP
31.18±1.76
48.50±2.12
36.05±1.94
34.76±1.97
35.01±1.94
37.79±1.85
34.57±1.97
30.14±1.64
33.55±1.90
36.15±2.03
33.06±1.84
36.14±1.93
29.27±1.71
33.46±1.95
89.08±0.70
21.88±5.58
44.30±0.61
GUT-Q-UP
5.77±2.20
18.07±2.34
90.83±0.80 -21.46±2.45
91.05±0.90
89.61±0.69
18.04±2.34
48.81±2.09
89.56±0.70
92.79±0.87 -20.53±2.39
52.88±2.03 58.58±2.08
-7.08±2.22
36.03±2.25
59.25±1.49
59.25±1.49
AUROC
MATH500
92.07±0.74 -26.70±2.38
89.74±0.87
20.77±2.37
90.85±0.68 -28.55±2.57 84.92±1.14
62.73±2.09 94.12±0.65 48.05±1.06
55.41±0.53
37.01±0.68
Eig-J
PRR
20.28±5.62
Deg-C
54.85±0.53
Eig-C
AUPRC 90.85±0.65 -28.55±2.57
93.87±0.66
62.43±2.13
62.64±2.11
46.53±2.16
51.34±2.05
51.70±2.05
53.03±2.00
36.29±1.89
37.84±1.92
55.11±1.68
42.46±1.61
31.13±1.96
50.80±1.48
54.85±2.01
54.77±2.04
50.00±0.00
50.00±0.00
54.28±2.06
53.48±1.93
43.78±1.45
43.78±1.44
64.15±2.23
56.47±1.98
52.99±1.98
37.36±2.06
56.18±1.45
56.18±1.43
AUROC
GSM8K
Deg-E
43.96±0.61
Eig-E
5.09±0.92
35.70±5.24
CoT-UQ-SAR
NS
32.96±5.39
CoT-UQ-MI
9.60±1.04
P(True)
35.43±5.25
CoCoA-MTE
CoT-UQ-ME
4.74±0.91
7.39±0.89
CoCoA-Ppl
2.01±0.91
41.65±0.69
ES
CoCoA-MSP
-3.61±0.92
30.24±0.78
SSAR
SD
20.35±0.90
-1.93±0.94
CSL
TSAR
-5.22±0.95
AS
5.87±0.83
17.36±0.89
FRD
48.37±0.58
17.28±0.89
RD
SAR
-62.21±1.25
CPMI
SE
6.28±0.94
-4.60±1.00
MPMI
31.97±0.71
MTE
MTLP
-11.17±1.04
Ppl
PRR
-11.17±1.04
MSP
UQ
27
86.84±0.79
-8.09±2.46
49.43±0.67
49.02±0.66
45.10±0.71
-9.26±1.11
53.52±0.68
-3.37±6.33
41.75±3.74
72.84±2.61
55.87±3.04
LS-R1
LS-R2
LS-RL
LS-B
KLE
LUQ
CoT-UQ-SP
Topo-UQ
GUT-Q-W
GUT-Q-H
71.38±1.45
81.68±1.29
64.77±1.59
51.38±1.90
28.97±1.40
52.32±1.75
31.05±1.64
29.30±1.58
29.25±1.62
33.21±1.60
30.82±1.77
27.78±1.44
33.20±1.66
32.69±1.59
33.30±1.62
4.12±2.48
33.21±4.80
28.22±5.11
75.36±1.24
41.59±2.06
6.65±2.47
29.07±2.40
35.65±2.34
26.35±2.42
2.72±2.54
49.68±2.25
32.90±2.10
92.73±0.63
95.46±0.49
89.94±0.79
46.49±4.35
54.59±4.13
11.12±4.69
85.11±1.12 -10.00±5.34
75.76±1.23
84.98±0.99
75.86±1.25
74.88±1.30
75.12±1.22
76.48±1.22
75.43±1.21
72.65±1.86
76.64±1.71
53.01±2.20
48.60±2.06
38.59±1.87
48.55±1.94
37.20±1.92
33.00±1.87
38.13±2.02
48.62±2.06
27.01±1.74
43.17±2.04
53.81±1.96
28.38±1.75
76.98±1.29 -11.87±2.53
76.33±1.27 50.95±2.16
52.22±2.05
28.84±1.73
50.61±2.00
34.47±1.98
3.98±2.42
50.42±2.17
2.81±2.51
59.51±1.92
50.00±0.00
63.32±1.90
61.24±1.98
47.95±2.03
74.77±1.24
77.73±1.27
76.53±1.26
74.93±1.27
41.07±2.02
53.71±2.08
54.15±2.01
52.97±2.04
29.81±1.82
AUPRC
PRR
27.33±5.00
13.68±4.96
25.50±4.85
41.38±5.13
41.40±5.15
6.27±5.13
19.41±5.17
51.98±4.70
0.54±4.92
39.79±4.90
32.58±4.96
49.49±2.21
61.89±2.42
61.35±2.55
62.26±2.48
40.59±1.83
26.38±9.91
20.98±9.99
58.01±4.40
58.55±4.34
22.42±3.84
50.00±0.00
61.74±4.62
59.23±4.51
61.27±4.54
27.79±4.44
-8.96±4.93
47.23±4.57
60.71±4.17
72.29±2.48
75.92±2.29
65.75±2.58
54.18±2.26
59.56±6.94
69.23±5.69
24.04±9.16
-5.71±9.67
49.99±2.44 -25.97±4.64
53.49±2.31
45.89±2.15
43.60±1.79
-0.49±4.89
-5.15±3.08
3.53±10.51
2.31±10.40
3.87±10.45
61.59±5.23
62.31±5.61
32.40±2.77
29.59±2.65
39.67±3.57 -26.40±3.39
51.53±5.97
48.81±6.23
51.27±5.95
25.47±4.96
24.69±3.39 -39.09±3.55
25.15±3.28 -35.62±3.43
25.03±3.23 -23.68±3.31
9.73±3.48
7.18±3.29
73.72±3.85
21.15±4.05
57.42±4.48
78.41±3.42
82.38±3.06
56.96±5.37
47.69±4.60
62.50±4.32
52.03±4.21
27.88±3.71
20.80±3.46
27.51±3.68
46.63±4.10
69.97±5.10
76.85±4.09
51.65±5.64
36.71±4.63
56.86±5.22
42.45±4.45
28.97±3.76
27.15±3.15
28.95±3.37
36.83±3.78
26.43±3.30
53.88±5.28
59.81±5.51
28.84±4.01
45.40±8.12
38.35±8.22
16.80±9.21
-3.37±6.33
33.40±2.55
-5.71±3.72
23.24±3.15
23.42±3.25
19.99±3.22
22.12±3.23
18.16±3.15
32.83±2.71
29.56±2.63
25.49±3.07
54.36±5.05 34.96±2.63
29.16±2.66
17.51±3.70
48.92±4.48
-6.89±3.38
24.76±3.08
63.76±4.19
46.66±2.02
-5.61±3.64 28.87±3.00 45.03±2.26
38.45±3.31
52.81±1.76
58.33±3.40
58.72±3.32
50.00±0.00
50.00±0.00
54.05±3.49
45.12±3.62
44.84±2.36
44.84±2.29
39.51±3.62
57.51±3.59
59.54±3.38
48.95±3.76
55.42±2.15
55.42±2.22
AUROC
FOLIO AUPRC
PRR
33.08±2.97
41.15±3.24
41.26±3.24
38.66±3.15
-2.18±3.43
-2.18±3.43
72.04±2.59
66.51±3.36
41.37±3.24
38.12±2.74
73.12±2.57 -14.36±3.63
75.20±3.05 -23.02±3.54
75.82±3.04 -16.65±3.57
71.92±2.43
71.92±2.58
69.76±3.50 -23.98±3.51
66.91±3.33
71.25±2.57
70.76±2.64
61.85±3.42
-4.94±3.41
1.26±3.62
77.15±2.83 74.47±3.12
8.92±3.15
69.82±3.31
74.25±2.55 -49.44±3.39
74.25±2.73 -49.44±3.39
33.42±2.24
35.73±2.95
56.38±2.14
61.96±3.02
60.09±2.77
49.33±0.37
49.33±0.40
65.69±2.86
39.78±2.80
33.46±2.15
33.48±2.14
38.05±2.89
54.72±3.11
57.39±2.91
43.28±3.08
66.52±2.12
66.52±2.12
AUROC
MMLU-Pro AUPRC
59.86±2.55
54.73±3.27
65.46±2.64
73.06±3.04
71.37±3.13
61.77±2.43
61.77±2.53
74.64±2.98
58.30±3.32
60.06±2.52
60.31±2.58
58.55±3.19
66.92±3.13
71.07±3.05
58.98±3.25
71.30±2.58
71.30±2.55
70.07±3.09
64.20±3.24
55.59±3.71
51.38±1.90
40.34±3.33
53.03±3.82
38.02±3.33
36.15±3.42
38.99±3.77
37.58±3.60
41.19±3.63
38.66±3.55
42.02±3.56
38.09±3.57
38.21±3.23
42.84±3.37
38.42±3.58
38.62±3.38
42.47±3.37
52.45±2.00
55.27±3.36
55.19±3.35
55.13±3.46
50.57±3.48
55.80±3.42
55.85±3.53
53.24±3.54
36.78±3.44
35.93±3.83
-0.21±3.37
47.91±2.73
17.24±3.21
30.20±2.83
49.75±2.68
48.71±2.43
3.32±3.43
6.86±3.43
19.20±8.03
22.41±7.70
21.42±7.75
8.19±3.55
-5.64±3.64
81.77±2.95
79.84±3.06
74.61±3.31
85.11±1.12
68.79±3.38
72.21±3.15
65.78±3.56
63.49±3.33
65.41±3.39
64.91±3.48
68.23±3.32
65.70±3.41
67.86±3.43
66.88±3.46
50.41±6.78
60.28±4.72
18.97±6.18
1.48±6.11
52.76±2.34
6.24±3.34
42.47±2.73
49.18±2.61
39.40±2.88
39.76±2.83
47.39±2.80
47.39±2.44
1.31±3.46
49.69±2.65
67.39±3.33 57.43±2.33
71.11±3.20
67.09±3.35
67.21±3.41
70.35±3.26
73.66±2.52
70.02±3.35
69.99±3.30
70.08±3.26
72.67±3.42
73.44±3.10
71.93±3.05 -17.76±3.56
70.69±3.37
58.39±3.36
60.36±3.50
72.81±2.49
75.51±2.45
58.70±3.04
45.82±2.76
30.43±2.68
45.11±3.05
32.91±2.82
28.77±2.63
32.12±2.77
32.47±2.78
28.34±2.75
34.13±2.75
48.60±3.09
28.00±2.68
26.27±2.46
42.44±3.11
27.72±2.70
33.03±2.72
49.71±3.20
46.12±1.71
53.79±2.96
54.46±2.90
54.14±2.96
51.53±3.03
57.78±3.06
61.36±3.05
57.21±3.30
27.95±2.74
46.70±3.00
79.03±3.07
82.90±2.46
68.52±3.18
27.76±2.56
52.03±3.03
60.30±3.06
51.57±2.87
50.33±2.97
50.89±3.04
51.69±3.07
49.54±2.93
53.50±3.16
62.26±3.33
49.00±2.90
48.47±2.94
60.44±3.28
48.92±2.91
52.35±3.09
63.58±3.35
61.19±2.58
74.29±2.88
75.24±2.73
74.94±2.75
65.94±3.28
69.42±3.46
72.07±3.08
68.10±3.40
50.89±2.92
64.52±3.13
55.40±5.35 -20.17±3.70 60.99±3.22 78.66±2.86 -38.73±3.56 68.53±2.73 75.78±2.93
35.57±3.68
28.42±3.61
50.38±5.25
24.80±3.28 -42.19±3.60
28.86±3.97
41.30±1.77 67.56±4.39 53.11±2.23
9.76±3.38
9.76±3.38
24.97±3.28 -41.13±3.62
41.03±3.70
41.03±3.58
76.03±3.74 64.78±5.59
21.35±3.85
73.40±3.98
-7.40±3.62
-6.70±3.38
-6.70±3.38
28.06±3.56 -40.60±3.54
30.58±3.69
39.18±4.33
39.16±4.02
7.26±3.59
43.94±5.05 -39.50±3.61 61.25±5.55
31.01±3.66
21.68±3.91
7.28±3.24
26.17±3.20 -37.74±3.37
28.18±3.57
61.06±5.18
21.17±3.68
PRR
52.35±4.41 -11.70±3.74
21.69±3.66
53.03±2.25 -28.36±4.20
56.34±2.47 -55.52±4.07
41.99±1.83
47.35±2.33 -11.14±4.93
56.64±2.54 -59.13±4.09
42.30±1.81
56.19±2.44 -54.14±4.79
AUPRC 52.35±4.27 -11.70±3.74
71.11±3.86
52.81±4.60
30.59±3.38
26.63±3.55
55.48±4.29
22.54±3.64
24.12±3.90
52.74±1.06
52.74±1.12
30.56±3.99
29.96±3.98
33.16±3.65
33.13±3.65
74.24±4.09
48.12±4.37
28.86±3.86
26.06±3.79
67.35±3.77
59.90±2.51 -60.23±4.79 76.71±4.02
54.80±1.80 -19.38±5.05
69.28±2.40
67.40±2.49
25.78±9.85
21.78±4.86
43.05±4.88
41.29±4.89
46.54±4.48
65.02±4.28
51.82±2.37 -56.66±3.58
61.69±2.39
59.86±2.55
44.61±1.95
61.98±2.32
62.28±2.60
62.72±2.56
55.47±1.70 -26.20±4.61
55.47±1.75 -26.20±4.61
60.42±2.51
57.56±2.42
55.79±1.94
55.70±1.97
57.15±2.37 -56.69±4.14
63.94±2.56 -12.39±4.79
62.99±2.53
37.77±4.71
60.85±2.01 -37.65±4.97 44.79±2.13
67.35±3.70
AUROC
AMC2022-2024
60.85±2.04 -37.65±4.97
33.48±4.82 63.45±1.94 69.40±2.43
26.47±2.49
12.23±2.47
9.62±2.48
17.21±2.47
35.57±2.45
77.54±1.31 -30.53±2.42
83.66±0.84
87.49±1.05
87.90±1.03
87.51±1.03
76.44±1.20
81.48±1.05
83.04±0.97
83.73±0.92
79.43±1.20
38.87±1.99
51.54±1.97
33.42±2.24
35.38±1.92
56.45±1.93
55.45±2.09
55.51±2.03
51.33±0.40
51.33±0.41
55.39±2.01
48.81±2.07
41.35±1.61
41.30±1.55
47.15±1.99
57.56±2.01
55.29±1.95
36.22±1.99
58.76±1.65
58.76±1.67
AUROC
MATH-500
Table 10: UQ performance comparisons for Qwen3-4B, where bold and underline denote the group best and second-best results, respectively.
81.68±1.83 86.61±1.02 96.93±0.36 61.93±3.59 80.13±1.57 79.45±2.07 80.82±4.24 89.48±2.19 84.60±3.43 53.19±8.16 73.46±2.79 84.65±2.73 69.23±4.61 81.83±2.26 86.38±2.33
42.64±0.70
Ecc-J
GUT-Q-UP
56.84±0.63
44.89±0.76
Ecc-C
43.13±0.72
44.92±0.73
60.76±0.63
Deg-C
Ecc-E
25.42±1.22
47.13±0.66
Deg-E
Deg-J
33.16±1.46
42.17±0.72
Eig-J
26.08±1.43
59.48±0.64
Eig-C
33.01±1.61
46.57±0.68
48.00±0.89
55.60±1.77
56.76±1.77
55.51±1.85
38.54±1.87
51.17±1.75
56.25±1.71
56.65±1.70
36.16±1.55
Eig-E
5.15±1.10
14.11±5.65
CoT-UQ-SAR
NS
17.07±5.68
CoT-UQ-MI
13.30±1.11
14.47±5.58
P(True)
CoT-UQ-ME
-27.79±1.15
-14.85±1.09
CoCoA-MTE
-30.10±1.20
CoCoA-MSP
CoCoA-Ppl
40.13±0.78
ES
35.85±2.39
10.34±2.40
79.16±1.15
33.10±0.78
37.76±1.49
-27.96±1.17 65.41±1.45 90.38±0.73
33.61±2.43
0.14±2.42
5.56±2.45
3.25±2.47
-4.91±2.52
-4.91±2.52
4.33±2.40
18.19±2.45
25.78±2.50
25.17±2.50
15.15±2.32
4.67±2.37
2.10±2.40
31.08±2.36
SD
75.20±1.28
85.13±0.79
85.45±0.88
84.21±0.99
84.31±0.81
84.31±0.79
87.09±0.88
83.69±0.97
83.88±0.83
83.88±0.89
80.60±1.17
87.55±0.86
84.56±0.91
74.56±1.26
PRR -8.09±2.46
SSAR
45.03±2.26
AUPRC 86.84±0.80
41.37±3.24
-6.89±3.38
TSAR
26.42±1.39
52.71±1.18
60.84±1.60
55.62±1.77
50.00±0.00
50.00±0.00
60.61±1.66
53.34±1.72
41.01±0.99
41.01±0.97
43.75±1.61
59.13±1.58
57.30±1.52
28.89±1.73
58.89±0.98
58.89±0.97
AUROC
GSM8K
72.04±2.59
-6.65±1.09
59.47±0.57
RAUQ-E
SAR
-40.55±1.22
RAUQ
SE
6.55±1.11
-29.39±1.18
MCNSE
6.55±1.11
MCSE
-39.67±1.17
14.53±1.08
RD
CSL
-11.23±0.99
CPMI
14.52±1.08
-28.86±1.13
MPMI
-17.30±1.19
-27.77±1.13
MTLP
AS
47.45±0.86
MTE
FRD
-33.66±1.22
Ppl
PRR
-33.66±1.22
MSP
UQ
28
PRR
PRR
8.25±2.45
9.95±2.44
9.95±2.45
72.48±0.96
72.48±0.98 -3.82±2.45
-3.82±2.45
74.45±1.35 -23.09±2.15
71.68±1.33
72.77±1.02
72.77±1.01
70.72±1.43 -22.03±2.52
75.98±1.26 -33.74±2.10
76.25±1.20 -24.19±2.16
20.06±2.50
73.41±0.96 -30.34±2.31 62.68±1.35
28.93±1.23
54.20±0.90
48.68±0.96
-1.41±1.40
50.04±0.93
LS-RL
LS-B
KLE
LUQ
GUT-Q-H
71.63±1.21
81.07±0.96
53.89±1.28
49.88±1.72
29.01±1.16
49.79±1.42
29.36±1.22
27.88±1.17
28.43±2.50
-9.07±2.55
15.06±4.91
15.00±4.84
14.20±4.84
8.37±2.52
4.56±2.51 51.36±2.32 51.86±1.99
46.79±2.35
45.20±2.36
41.59±2.37
85.32±0.96
90.40±0.82
77.92±1.15
71.64±1.54
61.01±1.34
49.96±3.91
51.54±3.77
8.35±5.98
-5.75±4.63
13.28±2.51
72.68±1.27 -10.33±2.48
60.79±1.30
59.27±1.20
59.92±1.30
41.66±2.43
74.58±1.76
72.78±1.82
52.76±2.09
49.31±1.60
44.90±2.20
53.50±2.01
25.65±1.83
24.82±1.76
26.84±1.76
27.16±1.91
23.96±1.80
59.94±1.27
46.41±2.07
49.81±2.02
24.41±1.72
41.00±1.91
12.06±2.52
61.22±1.32
-7.39±2.40
50.72±2.33
23.65±2.47
24.21±1.79
47.38±1.98
53.91±2.07
50.42±1.13
56.05±2.12
55.89±2.14
55.31±2.04
45.90±2.07
58.52±2.03
56.91±1.95
56.20±2.04
31.61±1.83
60.98±1.27 52.72±2.24
60.86±1.29
60.54±1.30
60.82±1.23
61.62±1.35 -10.62±2.32
60.56±1.30
61.20±1.35
61.95±1.35 -19.19±2.26
70.93±1.02
76.35±1.58
75.85±1.58
76.37±1.55
70.87±1.35
74.09±1.23 -25.98±2.14
73.15±1.30 -24.66±2.16
72.75±1.31 -18.10±2.29
61.33±1.29
36.44±1.96
19.85±4.76
31.22±1.84
55.32±5.36
39.43±2.88
30.81±4.70
59.56±4.49
29.80±4.75
30.12±4.80
50.54±0.44
50.54±0.47
38.99±4.54
30.36±4.44
37.48±2.99
37.59±2.92
64.41±4.66
52.55±4.65
26.68±4.69
38.49±4.74
62.78±2.99
62.78±3.01
AUROC
AUPRC
22.44±4.81
23.93±4.81
42.32±4.64
-9.05±4.26
-6.62±3.40 37.16±4.78
37.99±4.77
29.43±5.05
32.53±4.81
37.67±4.65
35.42±4.74
32.83±4.86
34.36±4.80
20.88±8.82
52.80±9.00 66.82±6.96
67.47±2.47 66.03±2.57
47.93±2.89 20.36±10.46
40.07±1.87
39.15±2.19 -14.39±3.80
44.13±2.53
29.01±1.51
28.71±1.61
29.63±1.66
29.81±1.67
28.57±1.62
40.66±2.27 -12.72±3.70
39.49±2.37 -13.64±3.20
28.66±1.65
38.68±2.12 -11.02±4.60
41.46±2.33 -14.13±3.33
28.63±1.53
40.92±2.20
42.52±2.38 -10.42±3.35
41.33±1.84
47.17±2.73 27.01±10.13
47.12±2.66 31.48±10.32
46.77±2.63 27.24±10.17
39.91±2.02 -42.80±3.54
45.32±2.57
42.90±2.57
43.67±2.76
31.80±1.61 47.63±4.81
81.60±4.18
70.88±5.48
50.49±4.18
53.94±4.12
56.93±5.25
36.53±3.80
27.32±4.21
26.80±4.18
29.09±4.24
28.00±4.16
31.48±4.14
56.48±4.88
56.67±4.50
26.54±4.27
53.86±5.24
58.17±4.74
26.84±4.06
55.88±4.60
56.78±4.71
51.18±1.84
63.04±5.17
63.64±5.21
62.64±5.47
66.68±4.18
28.46±4.98
27.98±4.91
22.91±4.66
28.93±4.31
PRR
-4.92±3.76
-9.48±3.76
27.57±3.79
27.29±3.80
31.03±3.88
73.49±2.95
56.76±1.29
56.76±1.26
70.11±3.17
50.91±3.22
26.17±1.89
26.20±1.89
28.02±2.90
72.59±2.90
69.64±2.92
52.59±3.36
73.98±1.80
73.98±1.98
AUROC
27.73±3.75
36.44±3.49
1.08±10.06
11.71±9.12
34.27±3.58
-8.26±3.73
20.11±3.61
38.19±3.50
24.23±3.62
24.87±3.60
16.54±3.73
33.71±3.58
13.65±3.77
22.06±3.65
41.38±3.52
20.06±3.69
22.14±3.65
37.92±3.53
17.27±3.74
67.48±6.39
57.95±6.90
54.37±6.06
50.23±6.11
34.12±6.66 -9.70±11.28
23.78±3.67
29.25±5.96
21.24±2.86
16.76±2.74
17.26±2.85
17.16±2.74
16.98±2.70
19.61±2.43
30.02±5.53
31.41±6.09
16.50±2.73
27.50±5.67
30.31±6.17
16.52±2.63
30.08±6.19
30.90±6.34
24.86±3.93 -12.38±3.84
37.26±6.59 -1.50±10.22
40.43±6.99
37.54±7.22 -1.73±10.22
28.21±5.38 -53.18±3.05
15.81±4.13 -56.99±3.41
15.91±4.66 -56.73±3.56
17.74±4.30 -52.16±3.61
19.09±2.91 46.82±3.53
43.60±3.43
38.61±6.10 -49.30±3.09
26.54±3.03
22.14±4.37
31.78±4.52 -30.98±3.14
PRR
-0.58±3.52
-0.70±3.37
13.82±3.41
17.61±3.66
17.61±3.66
50.38±3.16
33.40±3.33
45.81±3.21
34.29±3.46
37.29±3.33
41.65±3.33
37.71±3.18
-7.99±3.57
44.57±3.23
49.92±3.06
11.95±3.60
44.38±3.23
39.64±3.21
-9.09±3.59
8.73±3.59
5.72±7.93
13.05±7.98
6.05±8.07
1.18±3.29
41.58±3.16
74.01±2.72
70.02±4.20
67.81±4.14
56.20±4.61
45.91±5.64
6.31±8.05
37.51±3.07
48.49±3.70 -16.05±3.44
40.53±3.54
34.52±2.63
37.89±3.08
38.53±3.40
38.75±3.18
40.34±3.70
40.23±3.30
38.69±3.15
36.79±3.27
40.39±3.42
38.46±3.11
38.27±3.37
40.63±3.36
44.43±2.86
60.04±4.14
60.62±4.02
59.92±4.26
54.25±3.70
56.90±4.30 -19.23±3.31
56.63±4.29 -19.38±3.37
57.96±4.25 -18.82±3.33
29.42±2.48
73.90±2.44
68.57±2.77
49.69±3.01
48.72±3.02
36.51±2.85
56.04±2.92
37.28±2.84
31.21±2.70
34.84±2.83
36.03±2.76
32.63±2.87
38.76±3.00
52.47±3.02
30.53±2.73
30.96±2.87
49.70±2.99
30.67±2.59
37.42±3.03
54.46±3.00
48.50±2.06
53.98±3.01
57.67±3.11
54.36±3.10
47.47±3.09
61.48±2.78
60.88±2.73
60.57±2.98
27.85±2.42
33.13±2.83
37.00±3.33 51.14±3.34
37.62±1.75
42.14±2.87
67.34±2.55
17.44±3.64
29.41±3.44
53.38±1.75
67.65±2.77
67.46±2.83
50.37±0.31
50.37±0.29
68.15±2.80
44.68±3.01
37.45±1.77
37.45±1.79
53.09±2.88
66.91±3.71 -33.72±3.18
44.96±2.70
36.70±3.11
53.51±3.14 -14.08±3.42
67.76±4.41 -25.31±3.35
62.73±4.08 -24.69±3.35
48.94±2.89 -17.08±3.60
48.94±2.85 -17.08±3.60
54.71±3.79 -32.21±3.38
41.08±3.17
44.86±2.72
44.86±2.67
30.89±2.75
-6.03±7.02
76.21±2.63
65.68±2.85
48.77±3.02
62.75±1.71
62.75±1.70
AUROC
MMLU-Pro AUPRC
55.88±4.11
44.13±3.55
51.66±2.83
51.66±2.77
71.38±3.51
65.58±3.92
45.82±3.44
53.52±3.36
38.45±3.20
48.46±3.34
36.49±2.89
33.72±2.55
35.20±2.71
36.98±2.95
35.60±2.90
40.54±3.33
44.29±3.34
33.13±2.41
35.40±2.94
45.31±3.48
33.18±2.51
39.35±3.30
47.10±3.69
43.71±2.61
57.39±3.65
60.26±3.77
57.60±3.69
42.15±3.51
50.79±3.75
49.83±3.87
51.60±3.67
33.26±2.33
41.42±3.22
58.06±3.44
42.94±2.54
41.27±3.41
46.02±2.78
58.63±4.07
58.26±4.07
44.35±2.48
44.35±2.60
58.73±3.75
42.93±3.42
42.68±2.58
42.68±2.48
51.84±3.43
63.04±4.25 -32.32±3.18 69.65±2.87 65.61±3.73
61.77±4.24 -22.84±3.37
46.94±3.77
61.46±3.11 -41.57±2.80
61.82±3.67
53.36±3.31
AUPRC 61.46±3.11 -41.57±2.80
46.71±3.05 70.06±3.74
36.69±3.15
54.05±3.14
41.35±3.16
31.42±2.99
38.23±3.04
38.14±3.17
40.86±3.13
37.03±3.24
43.63±3.25
39.52±3.38
33.04±2.95
42.28±3.10
39.43±3.04
35.23±3.06
43.04±3.15
47.36±1.92
52.64±3.76
51.48±3.72
52.75±3.68
66.55±3.29
71.13±2.95
70.84±3.02
70.96±3.00
21.98±2.68
26.88±2.93
74.12±2.68
26.42±1.88
34.43±3.23
63.32±2.16
15.72±3.71 -62.93±3.37 74.68±2.84
15.80±3.79 -62.04±3.24
24.17±3.39 -18.80±3.32
24.17±3.24 -18.80±3.32
17.63±4.41 -55.74±3.62
16.36±3.36
22.28±3.26
22.32±3.13
39.88±5.73
24.20±6.23 -51.89±3.41
14.89±3.63 -49.12±3.47
20.46±4.01
29.86±3.90 -58.17±2.48
29.86±3.80 -58.17±2.48
38.13±2.22 -36.99±4.30 72.31±5.12 53.84±6.56
-0.10±4.95
12.60±4.67
33.60±4.88
48.31±2.06 -13.57±4.31
48.74±2.69
19.81±4.76
41.48±1.70 -17.57±4.11
52.15±2.32
19.87±2.42
-2.57±4.29
20.57±5.13
14.12±4.66
13.38±4.65
41.48±1.81 -17.57±4.11
46.87±2.57
42.57±2.49
40.91±1.78
40.91±1.75
43.89±2.41 -29.18±3.98
76.73±0.99 -29.91±2.24 64.53±1.72 65.90±1.25
18.23±4.43
51.02±2.79 -16.77±3.37
48.27±2.97
8.45±4.72
46.33±1.90 -31.58±3.21 30.65±1.80
40.89±1.69
30.81±1.99
61.32±1.40
62.66±1.98
PRR
FOLIO
Table 11: UQ performance comparisons for Qwen3-1.7B, where bold and underline denote the group best and second-best results.
77.50±1.83 85.48±0.86 92.83±0.67 63.39±3.36 80.68±1.50 74.08±2.45 68.00±6.93 82.90±3.72 68.00±6.30 62.40±5.75 80.83±2.37 74.55±4.10 64.10±4.21 78.51±2.28 75.99±3.21
69.63±2.27
52.54±2.62
GUT-Q-W
GUT-Q-UP
23.65±3.02
Topo-UQ
2.86±5.05
26.62±1.26
51.95±0.92
LS-R2
CoT-UQ-SP
27.73±1.18
51.76±0.92
LS-R1
29.51±1.24 31.24±1.28
49.30±0.92
42.13±1.03
30.65±1.29
Ecc-J
Ecc-E
27.76±1.14
31.03±1.24
28.98±1.22
28.34±1.13
31.32±1.28
45.84±0.92
56.85±1.60
56.31±1.67
57.19±1.57
50.18±1.39
56.61±1.36
53.64±1.38
53.62±1.46
31.81±1.36
35.16±1.23
58.04±1.18
Ecc-C
49.58±0.95
44.59±1.00
Deg-J
46.47±0.95
53.44±0.90
Deg-C
Eig-J
Deg-E
52.51±0.90
49.47±0.95
Eig-C
44.91±1.00
Eig-E
9.44±1.30
CoT-UQ-SAR
NS
15.62±4.65
17.58±4.58
CoT-UQ-MI
P(True)
17.64±4.63
-16.90±1.46
CoCoA-MTE
CoT-UQ-ME
-20.96±1.47
-25.58±1.46
CoCoA-Ppl
41.14±1.11
-20.93±1.46
CoCoA-MSP
31.36±1.16
SD
ES
-17.18±1.45
SSAR
AUPRC
AMC2022-2024
46.33±1.97 -31.58±3.15
64.35±1.96 53.28±2.92
50.17±0.14
50.17±0.15
58.78±2.03
46.80±2.07
41.01±1.09
41.02±1.05
56.13±2.01
62.85±1.92
59.69±1.90
31.66±1.95
59.01±0.98
59.01±1.03
AUROC
MATH-500
41.00±1.05
9.93±2.44
35.30±2.40
61.78±1.32
8.49±1.36
71.48±1.03
28.64±1.19
54.11±0.91
SAR
TSAR
47.64±0.54
73.03±1.01 -30.48±2.04
51.32±0.69
78.58±1.20 -31.07±2.10
-3.86±1.46
SE
-36.08±1.50 61.39±1.37
RAUQ-E
50.00±0.00
AUPRC 73.41±1.01 -30.34±2.30
61.37±1.35 78.60±1.19 -32.57±2.08
-0.49±1.37
-36.39±1.51
RAUQ
50.00±0.00
56.07±1.48
48.52±1.39
47.75±0.56
47.75±0.53
50.47±1.50
58.87±1.43
57.79±1.33
35.08±1.40
52.25±0.55
52.25±0.52
AUROC
GSM8K
MCNSE
-0.49±1.37
CSL
MCSE
2.85±1.34
-30.05±1.50
AS
CPMI
8.38±1.36
-16.29±1.43
MPMI
FRD
-33.59±1.51
MTLP
8.38±1.36
-26.91±1.48
MTE
RD
-8.80±1.44
35.50±1.23
Ppl
-8.75±1.44
MSP
UQ
29
-0.19±1.44
MTLP
MPMI
GSM8K
53.98±1.39
48.87±1.37
29.43±1.31
49.90±0.72
49.90±0.68
AUROC
14.14±1.42
CoCoA-MTE
CoT-UQ-SAR
-7.43±1.63
-1.09±1.88
-7.07±1.88
-7.07±1.88
-1.80±1.61
-1.61±1.56
9.74±2.31
9.28±4.92
-3.83±1.90
-8.56±1.48
-6.57±1.54
-3.45±1.47
39.74±1.34 35.92±1.24
28.62±1.50
24.87±1.58
17.91±1.59
25.04±1.58
26.48±1.59
20.07±1.56
-6.89±1.44
32.52±1.50
-2.97±7.64
5.55±2.90
45.88±2.68
35.39±2.83
Ecc-C
Ecc-J
LS-R1
LS-R2
LS-RL
LS-B
KLE
LUQ
CoT-UQ-SP
Topo-UQ
GUT-Q-W
GUT-Q-H
39.83±1.37
8.83±2.28
11.49±2.35
-9.52±2.04
9.79±2.36
13.22±2.31
20.11±2.39
17.81±2.37
16.11±2.34
46.22±2.08
53.41±2.14
31.02±1.69
37.94±3.79
40.24±4.79
42.51±4.84
2.02±4.96
6.46±5.03
22.50±1.14 -11.75±1.87
30.39±1.29 -14.69±1.80
24.25±1.04
23.03±0.99
23.50±0.98
25.15±1.12
23.34±1.06
22.68±1.22 -14.26±1.75
21.52±1.06 -10.59±1.83
25.92±1.11
22.99±1.32
22.90±1.23 -14.07±1.81
26.86±1.27
23.72±1.27 -13.01±1.79
23.23±1.15 -14.72±1.73
14.45±1.36
72.53±2.30
71.36±2.58
47.67±2.80
52.65±1.89
54.76±2.87
55.01±2.66
38.83±4.16
56.64±1.31
7.55±2.23
4.97±2.37
5.50±4.50
5.78±1.64
46.43±0.99
48.01±9.78
-11.85±5.46
-11.14±7.03
3.36±2.04
-2.79±3.25
-9.72±3.79
8.47±2.35
7.44±1.89
9.14±2.38
5.73±1.84
9.12±3.32
7.11±5.32
10.40±7.59
FOLIO
54.44±3.20
52.85±3.24
54.16±3.25
50.56±0.91
50.57±0.91
AUROC
-1.70±3.62
14.57±7.48
4.59±7.10
-7.39±7.97
-1.45±7.52
21.11±3.61
1.74±3.69
7.68±3.79
8.01±3.82
7.79±3.79
11.72±3.78
47.04±3.23
55.65±3.24
55.86±3.13
43.80±3.71
48.32±3.10
45.77±3.28
51.92±3.27
48.76±3.29
47.52±3.16
48.88±3.45
47.53±3.34
49.13±3.41
46.27±3.30
47.25±3.17
50.28±3.34
44.63±3.31
47.62±3.36
50.30±3.24
44.21±3.40
48.74±3.41
48.06±3.33
50.19±3.19
51.64±3.24
50.47±3.29
1.48±6.11
17.99±3.33
3.59±3.11
19.68±3.35
49.41±4.15
45.34±3.89
13.64±7.16
33.29±6.64
44.38±3.71 -25.56±10.82
40.06±3.24
43.57±3.98
45.01±3.66
45.74±4.15
AUPRC
27.76±2.56
25.14±2.99
25.45±2.51
24.57±2.83
22.44±2.44
22.64±2.51
25.23±2.93
23.98±2.70
26.79±3.05
25.03±2.79
24.49±2.96
25.85±3.00
25.76±3.07
24.23±2.89
26.95±3.31
26.26±2.90
25.29±2.53
30.62±3.22
31.95±3.36
30.53±3.31
23.80±3.35
29.04±3.32
27.19±3.37
26.78±3.34
21.57±3.64
21.41±2.74
28.37±2.81
31.35±3.03
24.30±2.65
28.25±2.51
32.99±3.97
33.62±3.89
27.92±2.35
27.92±2.34
30.14±3.48
30.21±3.91
30.07±2.97
29.97±2.82
27.36±3.43
32.83±3.97
31.80±4.19
27.34±3.42
26.58±2.35
26.58±2.30
54.33±3.49
65.50±3.22
36.16±4.23
46.20±4.91
44.72±3.54 42.95±3.90
45.82±2.76
42.05±3.22
44.41±3.17
41.94±3.23
37.17±3.28 37.64±3.28
40.76±3.37
40.15±3.30
46.42±3.42
42.77±3.33
39.32±3.28
43.71±3.42
45.36±3.24
39.67±3.37
44.80±3.34
45.85±3.35
42.94±2.86
50.05±3.26
53.44±3.17
50.02±3.15
49.56±3.19
52.52±3.49
26.70±3.31
24.41±3.30
21.79±3.35
15.40±3.31
10.89±3.13
22.25±3.29
21.14±3.32
4.42±3.12
21.67±3.26
19.72±3.32
4.05±3.10
7.52±3.20
-2.14±6.31
1.37±6.07
-2.39±6.29
-13.39±3.30
-9.47±3.17
51.53±3.18
51.00±3.28
37.92±3.14
40.31±3.10
50.81±3.08
53.43±1.73
43.07±3.26
50.79±1.02
57.90±3.30
43.57±3.79 28.37±3.30
45.35±4.03
44.75±4.07
44.25±3.99
42.36±3.75
40.55±3.51
45.77±3.91
42.72±3.93
42.77±3.94
46.05±4.13
42.28±4.06
43.75±3.96
43.50±3.53
43.03±3.89
41.92±3.79
43.19±3.82
41.86±3.61
49.11±3.87
-9.44±3.18
8.57±3.09
10.72±3.21
1.31±3.29
-1.91±3.03
12.32±3.29
-1.46±3.10
-15.31±3.14
-7.58±3.19
52.03±3.17
50.00±0.00
50.00±0.00
54.14±3.29
50.67±3.38
53.33±1.67
53.32±1.66
54.36±3.30
57.33±3.16
56.93±3.30
46.66±3.35
46.69±1.74
46.69±1.68
AUROC
MMLU-Pro
-16.24±3.19 58.17±3.38
-0.02±3.08
-0.02±3.08
-11.37±3.02
-2.87±3.35
-1.85±3.04
-1.84±3.04
49.52±3.91
40.16±3.27
43.08±3.27
43.62±2.99
42.83±2.75
40.74±3.60
42.58±2.79
53.95±3.93
48.38±3.80
42.86±2.75
42.86±2.73
48.07±4.26
45.79±3.94
43.79±2.84
43.79±2.82
-15.94±3.04
-14.12±3.12
-17.83±3.06
5.45±3.16
11.28±3.36
11.28±3.36
PRR
53.62±3.36 54.08±3.83
51.43±3.33
41.68±3.08
42.53±2.94
51.14±2.44
49.42±0.97
45.46±3.16
49.43±0.47
56.42±3.43
52.87±3.18
50.00±0.00
50.00±0.00
51.89±3.24
51.05±3.16
49.46±0.92
49.46±0.93
48.36±3.73
49.15±3.77
49.70±3.68
46.30±4.16
43.14±2.74
43.15±2.86
AUPRC
Table 12: UQ performance comparisons for Qwen3-0.6B, where bold and underline denote the group best and second-best results.
15.48±4.79 31.43±6.96 62.18±3.26 57.75±4.49 37.18±6.33 66.86±3.11 48.92±4.53
12.65±3.54
58.51±nan 19.48±8.36 62.81±nan 14.75±8.38 67.52±nan
5.38±6.37
44.88±6.09
46.43±0.98
49.67±6.21
31.31±nan
47.79±8.77
-0.61±3.77 43.36±10.21
-0.99±3.93
8.54±9.82
4.14±3.49
16.04±9.11
42.58±3.98
-8.40±3.53 54.42±10.41
19.76±3.72
61.84±7.85 25.22±6.07
2.40±3.77
23.28±3.67
14.87±3.65
2.42±3.78
23.67±3.67
15.22±3.60
10.03±3.70
1.84±7.32
-1.08±7.11
-12.60±2.74
9.93±4.32
14.25±3.36
7.35±2.26
10.08±4.85
20.12±3.91
8.73±2.60
6.61±2.06
7.44±1.89
7.44±1.89
2.29±7.32
15.76±3.75
9.17±3.78
54.31±6.19
34.96±8.44
47.01±7.98
48.56±5.67
37.06±8.31
51.44±7.09
53.65±5.62
45.13±1.07
46.43±0.99
7.44±1.88
7.87±3.61
5.12±3.62
4.98±3.59
17.53±3.75
9.49±2.57
-5.86±1.92
12.86±3.63
3.89±2.59
2.77±2.00
10.73±3.62
-4.13±2.22
-3.66±1.75
-12.78±2.87
-11.93±6.95
-11.93±6.95
-1.99±3.94
8.62±4.71 26.40±3.83
4.53±3.29
14.12±4.65
46.43±0.97
10.82±3.67
15.39±3.79
0.25±3.77
6.02±3.71
12.55±3.79
12.55±3.79
0.00±3.56
3.21±3.88
6.61±2.08 -13.26±3.85
5.05±3.00
4.95±2.23
4.86±2.64
4.77±3.07
6.61±1.99
6.61±1.96
5.68±9.17
5.77±2.68
6.61±1.99 -13.29±3.85
6.61±1.96 -13.29±3.85
-15.85±3.80 63.05±8.78 -11.93±6.95
-0.39±3.73
5.67±3.75
0.42±3.70
5.99±3.92
5.71±3.92
PRR
7.43±2.11 -17.04±3.79 57.01±3.25
11.83±4.06
5.89±2.96
4.91±2.08
4.97±2.46
29.31±6.49
28.87±8.66
24.50±6.09
51.00±7.90
21.57±7.59
43.36±1.31
30.86±8.46
32.08±5.36
27.10±6.39
25.77±6.50
50.00±nan
50.00±0.00
37.50±9.31
30.64±7.68
43.36±1.26
43.36±1.27
52.32±5.25
44.25±8.18
39.16±7.61
28.43±6.75
7.55±2.31
AUPRC
29.09±7.00
2.07±2.51
1.68±2.53
23.92±3.33
35.29±4.07
-8.65±2.27
21.47±2.50
-10.50±2.87
6.55±4.60
16.86±2.34
41.17±4.31 17.35±13.68
18.82±2.57
14.75±1.60
17.08±2.81
16.63±2.29
11.51±1.33
10.18±1.29
36.69±2.49
10.46±1.23
31.66±2.64
10.71±1.29
12.05±1.09
16.97±2.56
15.37±1.95
11.19±1.16
17.41±3.29
17.22±2.57
11.37±1.12
18.41±3.47
17.34±2.72
14.70±1.67
15.34±1.91
15.18±1.79
15.19±1.88
17.77±1.66
17.09±1.75
17.19±1.64
16.34±1.70
15.18±1.83
16.08±1.40
19.88±2.59
33.63±2.68
34.95±2.70
38.74±2.51
55.66±2.69
51.88±2.56
35.79±2.55
54.16±2.98
56.16±2.60
36.90±2.58
55.35±2.76
55.88±2.65
50.98±1.39
52.26±1.65
52.05±1.66
52.04±1.60
52.30±2.60
52.23±2.41
50.84±2.38
49.25±2.48
39.98±2.77
17.34±1.61 11.15±1.62
52.02±2.51 75.31±1.20 57.57±2.06 52.04±4.58 76.50±2.31 48.71±4.35
68.25±1.27
71.67±1.31
50.26±1.40
49.11±3.55
33.46±1.41
50.98±1.29
38.75±1.30
34.70±1.29
36.01±1.26
34.92±1.34
30.85±1.29
19.86±1.57
35.59±1.57
34.23±1.42
34.66±1.40
41.27±1.38
36.28±1.43
35.63±1.35
27.54±1.08 -12.16±1.89
9.17±4.88
45.79±4.60
29.69±1.43
24.36±1.18
26.31±1.61
27.66±1.71
20.04±2.29
39.86±2.51
26.90±1.16
25.77±1.20 20.15±2.12
48.60±1.69 59.24±2.64
10.29±5.05
45.12±1.04
58.96±1.55 37.76±2.64
28.86±1.41 -17.30±1.73
29.23±1.01 -15.46±1.92
20.72±0.95
29.67±1.02 -24.96±1.39
5.13±2.40
7.27±2.40
23.35±2.03
-8.89±2.69
-8.89±2.69
-1.82±3.16
11.01±2.72
-10.50±2.87
-10.50±2.87
-16.54±2.00
-2.84±3.08
9.65±2.62
2.71±2.78
-20.88±1.81
56.64±1.30
AUROC
AMC2022-2024
-20.88±1.81
PRR
14.60±1.33
14.60±1.38
18.75±1.70
20.14±1.80
14.32±1.31
14.39±1.36
16.74±1.44
21.34±2.05
23.16±3.38
10.96±1.66
15.07±1.55
15.07±1.61
AUPRC
60.88±2.73 24.68±3.27
50.00±0.00
50.00±0.00
48.86±2.58
54.81±2.50
48.49±1.73
48.55±1.74
49.20±2.59
56.76±2.61
57.65±2.62
36.46±2.53
51.67±1.62
51.67±1.65
AUROC
MATH-500
33.12±1.71 -16.41±1.45 61.71±2.49
28.14±1.30 -14.71±1.41
29.95±1.02
29.95±1.02
31.90±1.38
31.99±1.54
31.47±1.18 -15.54±1.94
31.55±1.15 -15.53±1.93
36.65±1.55
33.87±1.76 -10.62±1.56
28.58±1.44
0.59±1.58
29.94±1.07 -15.13±1.43 20.75±1.03
44.04±4.49
Ecc-E
GUT-Q-UP
PRR
29.94±1.07 -15.13±1.43
AUPRC
56.64±3.49 46.88±4.75
54.51±3.48
55.30±3.49
47.65±1.38
39.52±1.41
43.86±1.35
46.44±1.32
41.65±1.33
40.61±1.40
47.23±1.37
Deg-J
29.84±1.54
32.75±1.47
16.80±1.57
Eig-J
Deg-C
29.51±1.48
Eig-C
Deg-E
27.77±1.55
Eig-E
4.09±1.39
9.54±7.79
16.08±7.74
CoT-UQ-MI
NS
13.54±7.84
CoT-UQ-ME
7.55±1.59
8.15±1.43
CoCoA-Ppl
P(True)
3.22±1.40
CoCoA-MSP
11.48±1.49
17.16±1.48
5.11±1.47
SSAR
ES
-3.73±1.40
TSAR
SD
28.14±1.31 49.81±0.70
49.33±0.62
3.76±1.43
SE
54.29±1.38
40.66±1.55
-6.06±1.37
RAUQ-E
46.78±1.31
50.00±0.00
50.00±0.00
53.87±1.31
54.07±1.38
50.16±0.70
50.17±0.71
SAR
1.71±1.42
MCSE
2.65±1.37
1.71±1.42
CSL
RAUQ
-6.81±1.36
AS
MCNSE
-3.94±1.39
-10.32±1.34
FRD
-3.94±1.39
RD
-14.31±1.41 58.42±1.35
5.87±1.44
MTE
CPMI
0.05±1.42
29.11±1.54
Ppl
0.05±1.42
PRR
MSP
UQ
C. Additional Experimental Details on GUT-O
This appendix provides additional implementation details and experimental results on the proposed GUT-O module.
C.1. Implementation Details The GUT-O was implemented with the open-r112 repository. Table 13 lists the configuration of hyperparameters employed in the GUT-O. We fine-tuned the Qwen3-8B, Qwen3-4B, Qwen3-1.7B, and Qwen3-0.6B on the training set of the GSM8K [6], MATH [18], FOLIO [16], MMLU-Pro [45], and AMC2022-2024, respectively, where we randomly select 20% instances of the synthetic AMC2022-2024 as training instances. See Appendix B.1 for details of AMC2022-2024. Hyperparameter
Value
Hyperparameter
Value
Batch Size
128
Max New Tokens
3072
Group Size M
7
Learning Rate
3 × 10−6
KL Penalty β
0.0005
Learning Rate Scheduler
Cosine
Temperature T
1.0
Warmup Ratio
0.1
Clip Ratio ϵ
0.2
Optimizer
AdamW
Training Steps
60
Table 13: Configuration of hyperparameters for GUT-O.
C.2. UO Performance Evaluation Table 14 shows the percentage changes in mean uncertainty and absolute changes in accuracy for Qwen31.7B and Qwen3-0.6B. It is observed that the proposed GUT-O effectively reduces the uncertainty and improves accuracy across LLM scales, datasets, and UQ evaluation metrics. This observation validates the proposed GUT-O module.
12 https://github.com/huggingface/open-r1
30
Table 14: Comparisons of UO performance for Qwen3 families across 5 reasoning datasets, where ‘+’ indicates the applied fine-tuning method, and values in parentheses indicate the percentage change in mean uncertainty. Mean Uncertainty Dataset
Models
Accuracy
MTLP GUT-Q-W
Qwen3-0.6B 28.89 ± 0.97 −7.11 ± 0.02 3.10 ± 0.02
4.14 ± 0.04
5.77 ± 0.04 (-2.7%)
3.99 ± 0.04 (-3.6%)
+GUT-O 38.06 ± 1.07 −8.54 ± 0.02 2.78 ± 0.02 (-10.3%) 5.27 ± 0.04 (-11.1%)
3.32 ± 0.03 (-19.8%)
Qwen3-1.7B 71.11 ± 1.00 −14.54 ± 0.03 1.68 ± 0.02
5.51 ± 0.06
1.58 ± 0.03
+GRPO 72.93 ± 0.95 −14.28 ± 0.04 1.69 ± 0.02 (+0.6%) 5.41 ± 0.06 (-1.8%)
1.62 ± 0.03 (+2.5%)
+GUT-O 72.78 ± 0.96 −15.37 ± 0.04 1.58 ± 0.02 (-6.0%)
5.22 ± 0.06 (-5.3%)
1.51 ± 0.02 (-4.4%)
4.47 ± 0.04
1.18 ± 0.02
+GRPO 86.81 ± 0.75 −13.22 ± 0.04 1.41 ± 0.02 (-2.8%)
4.42 ± 0.04 (-1.1%)
1.19 ± 0.02 (+0.8%)
+GUT-O 84.91 ± 0.80 −13.49 ± 0.03 1.39 ± 0.02 (-4.1%)
4.37 ± 0.04 (-2.2%)
1.15 ± 0.02 (-2.5%)
4.15 ± 0.03
0.95 ± 0.01
Qwen3-4B 84.08 ± 0.79 −13.48 ± 0.04 1.45 ± 0.02
Qwen3-8B 89.61 ± 0.67 −16.56 ± 0.05 1.28 ± 0.01
+GRPO 90.75 ± 0.64 −15.96 ± 0.05 1.28 ± 0.01 (+0.0%) 4.14 ± 0.03 (-0.2%)
0.94 ± 0.01 (-1.1%)
+GUT-O 89.76 ± 0.64 −17.20 ± 0.05 1.25 ± 0.01 (-2.3%)
0.92 ± 0.01 (-3.2%)
Qwen3-0.6B 18.00 ± 1.31 −7.55 ± 0.04 0.77 ± 0.01
4.14 ± 0.04 (-0.2%) 1.86 ± 0.03
1.14 ± 0.01
1.88 ± 0.03 (+1.1%)
1.17 ± 0.02 (+2.6%)
+GUT-O 21.40 ± 1.43 −8.94 ± 0.04 0.64 ± 0.01 (-16.9%) 1.51 ± 0.03 (-18.8%)
0.88 ± 0.01 (-22.8%)
+GRPO 23.40 ± 1.51 −7.67 ± 0.04 0.74 ± 0.01 (-3.9%)
Qwen3-1.7B 42.20 ± 1.79 −13.45 ± 0.06 0.43 ± 0.01
MATH-500
GUT-Q-UP
5.93 ± 0.04
+GRPO 38.97 ± 1.05 −7.38 ± 0.02 3.03 ± 0.02 (-2.3%)
GSM8K
GUT-Q-H
2.03 ± 0.05
0.47 ± 0.01
+GRPO 46.40 ± 1.81 −13.56 ± 0.07 0.42 ± 0.01 (-2.3%)
1.89 ± 0.05 (-6.9%)
0.46 ± 0.01 (-2.1%)
+GUT-O 43.00 ± 1.67 −14.03 ± 0.07 0.39 ± 0.01 (-9.3%)
1.82 ± 0.05 (-10.3%)
0.43 ± 0.01 (-8.5%)
Qwen3-4B 53.80 ± 1.71 −14.14 ± 0.08 0.37 ± 0.01
1.76 ± 0.05
0.40 ± 0.01
+GRPO 61.00 ± 1.68 −14.11 ± 0.09 0.38 ± 0.01 (+2.7%) 1.78 ± 0.04 (+1.1%)
0.40 ± 0.01 (+0.0%)
+GUT-O 57.00 ± 1.71 −14.25 ± 0.07 0.34 ± 0.00 (-8.1%)
1.70 ± 0.04 (-3.4%)
0.32 ± 0.01 (-20.0%)
1.46 ± 0.04
0.24 ± 0.01
Qwen3-8B 64.20 ± 1.68 −15.14 ± 0.09 0.33 ± 0.01
+GRPO 66.60 ± 1.64 −14.97 ± 0.09 0.33 ± 0.01 (+0.0%) 1.43 ± 0.04 (-2.1%)
0.25 ± 0.01 (+4.2%)
+GUT-O 64.80 ± 1.72 −15.78 ± 0.08 0.31 ± 0.01 (-6.1%)
1.38 ± 0.04 (-5.5%)
0.23 ± 0.01 (-4.2%)
14.14 ± 0.41
5.72 ± 0.14
Qwen3-0.6B 9.09 ± 2.09
−7.12 ± 0.07 5.10 ± 0.10
+GRPO 11.57 ± 2.32 −7.22 ± 0.08 4.99 ± 0.10 (-2.2%) +GUT-O 9.09 ± 2.11
15.07 ± 0.38 (+6.6%) 6.37 ± 0.18 (+11.4%)
−8.51 ± 0.09 3.97 ± 0.09 (-22.2%) 10.34 ± 0.30 (-26.9%) 4.16 ± 0.12 (-27.3%)
Qwen3-1.7B 22.31 ± 3.03 −12.96 ± 0.15 2.92 ± 0.08
20.35 ± 0.94
3.54 ± 0.14
+GRPO 30.58 ± 3.31 −12.98 ± 0.14 3.10 ± 0.08 (+6.2%) 17.69 ± 0.88 (-13.1%) 3.60 ± 0.16 (+1.7%) AMC2022-2024
+GUT-O 24.79 ± 3.25 −14.08 ± 0.15 2.46 ± 0.07 (-15.8%) 16.16 ± 0.74 (-20.6%) 2.80 ± 0.13 (-20.9%) Qwen3-4B 39.67 ± 3.52 −13.53 ± 0.15 2.69 ± 0.08
16.88 ± 0.86
3.27 ± 0.17
+GRPO 42.98 ± 3.73 −13.92 ± 0.16 2.74 ± 0.08 (+1.9%) 15.19 ± 0.67 (-10.0%) 3.07 ± 0.15 (-6.1%) +GUT-O 44.63 ± 3.62 −14.45 ± 0.14 2.16 ± 0.07 (-19.7%) 14.66 ± 0.75 (-13.2%) 1.67 ± 0.09 (-48.9%) Continued on next page
31
Table 14: Comparisons of UO performance for Qwen3 families (Continued). Mean Uncertainty Dataset
Models
Accuracy
MTLP GUT-Q-W
Qwen3-8B 48.76 ± 3.84 −15.32 ± 0.19 2.30 ± 0.07
1.58 ± 0.10
+GRPO 51.24 ± 3.71 −14.90 ± 0.17 2.37 ± 0.08 (+3.0%) 15.94 ± 0.77 (-6.2%)
1.75 ± 0.10 (+10.8%)
15.87 ± 1.00 (-6.6%)
1.43 ± 0.09 (-9.5%)
1.49 ± 0.03
0.46 ± 0.01
+GRPO 51.23 ± 2.73 −7.23 ± 0.04 0.46 ± 0.01 (-2.1%)
1.34 ± 0.03 (-10.1%)
0.44 ± 0.01 (-4.3%)
+GUT-O 46.31 ± 2.78 −7.19 ± 0.04 0.46 ± 0.01 (-2.1%)
1.38 ± 0.02 (-7.4%)
0.45 ± 0.01 (-2.2%)
Qwen3-1.7B 43.84 ± 2.98 −13.46 ± 0.09 0.24 ± 0.01
1.31 ± 0.04
0.25 ± 0.01
+GRPO 48.77 ± 2.97 −13.15 ± 0.08 0.24 ± 0.01 (+0.0%) 1.33 ± 0.04 (+1.5%)
0.25 ± 0.01 (+0.0%)
+GUT-O 45.81 ± 3.04 −14.43 ± 0.08 0.21 ± 0.00 (-12.5%) 1.08 ± 0.03 (-17.6%)
0.23 ± 0.01 (-8.0%)
Qwen3-4B 68.47 ± 2.56 −10.49 ± 0.07 0.31 ± 0.01
1.10 ± 0.03
0.31 ± 0.01
1.08 ± 0.03 (-1.8%)
0.30 ± 0.01 (-3.2%)
+GUT-O 68.97 ± 2.53 −11.01 ± 0.06 0.27 ± 0.01 (-12.9%) 1.05 ± 0.03 (-4.5%)
0.28 ± 0.01 (-9.7%)
+GRPO 72.91 ± 2.54 −10.52 ± 0.06 0.29 ± 0.01 (-6.5%)
Qwen3-8B 72.41 ± 2.45 −12.75 ± 0.15 0.27 ± 0.01
1.03 ± 0.03
0.25 ± 0.01
+GRPO 74.88 ± 2.44 −13.18 ± 0.16 0.27 ± 0.01 (+0.0%) 0.98 ± 0.03 (-4.9%)
0.25 ± 0.01 (+0.0%)
+GUT-O 74.88 ± 2.48 −13.31 ± 0.16 0.25 ± 0.01 (-7.4%)
0.24 ± 0.01 (-4.0%)
Qwen3-0.6B 29.17 ± 2.39 −6.19 ± 0.04 0.87 ± 0.02
0.97 ± 0.03 (-5.8%) 1.68 ± 0.03
0.78 ± 0.01
1.73 ± 0.03 (+3.0%)
0.78 ± 0.01 (+0.0%)
+GUT-O 29.58 ± 2.30 −7.31 ± 0.05 0.75 ± 0.02 (-13.8%) 1.31 ± 0.03 (-22.0%)
0.65 ± 0.01 (-16.7%)
+GRPO 33.33 ± 2.39 −6.21 ± 0.05 0.85 ± 0.02 (-2.3%)
Qwen3-1.7B 43.33 ± 2.65 −9.86 ± 0.09 0.58 ± 0.01
MMLU-Pro
16.99 ± 1.00
GUT-Q-UP
+GUT-O 49.59 ± 3.64 −16.11 ± 0.20 2.24 ± 0.07 (-2.6%) Qwen3-0.6B 44.83 ± 2.69 −7.04 ± 0.04 0.47 ± 0.01
FOLIO
GUT-Q-H
2.31 ± 0.06
0.54 ± 0.01
+GRPO 50.42 ± 2.55 −9.90 ± 0.09 0.58 ± 0.01 (+0.0%) 2.32 ± 0.06 (+0.4%)
0.54 ± 0.01 (+0.0%)
+GUT-O 46.25 ± 2.60 −10.61 ± 0.09 0.53 ± 0.01 (-8.6%)
1.88 ± 0.05 (-18.6%)
0.48 ± 0.01 (-11.1%)
1.50 ± 0.04
0.36 ± 0.01
+GRPO 67.08 ± 2.47 −11.04 ± 0.08 0.45 ± 0.01 (-2.2%)
1.50 ± 0.04 (+0.0%)
0.36 ± 0.01 (+0.0%)
+GUT-O 62.08 ± 2.49 −11.57 ± 0.08 0.45 ± 0.01 (-2.2%)
1.27 ± 0.04 (-15.3%)
0.31 ± 0.01 (-13.9%)
1.33 ± 0.04
0.30 ± 0.01
Qwen3-4B 60.83 ± 2.60 −11.05 ± 0.09 0.46 ± 0.01
Qwen3-8B 67.92 ± 2.49 −12.79 ± 0.14 0.43 ± 0.01
+GRPO 70.42 ± 2.42 −12.61 ± 0.15 0.44 ± 0.01 (+2.3%) 1.33 ± 0.04 (+0.0%)
0.30 ± 0.01 (+0.0%)
+GUT-O 69.58 ± 2.39 −13.53 ± 0.16 0.41 ± 0.01 (-4.7%)
0.29 ± 0.01 (-3.3%)
1.23 ± 0.04 (-7.5%)
C.3. Validation of the MTLP optimization target Table 15 shows the PCCs between the GUT-Q-derived uncertainty U (x) and the sum of the MTLP PN optimization target i=1 U (x, ci )/N for the Qwen3 family, at scales of 0.6B, 1.7B, 4B, and 8B, across five investigated datasets. It is observed that the U (x) generally has significant positive PCCs with the MTLP optimization target spanning datasets for all 4 LLMs. Therefore, we can consider that the MTLP serves as a valid optimization target for optimizing U (x). 32
Model
UQ
GSM8K
MATH-500
AMC2022-2024
FOLIO
MMLU-Pro
GUT-Q-W
0.2755 ± 0.0203 0.1611 ± 0.0314
0.3310 ± 0.0646
0.2397 ± 0.0548 0.2566 ± 0.0432
Qwen3-0.6B GUT-Q-H
0.2033 ± 0.0220 0.1718 ± 0.0345
0.0724 ± 0.0690
0.1917 ± 0.0484 0.2542 ± 0.0449
GUT-Q-UP 0.2980 ± 0.0207 0.0734 ± 0.0337
0.1034 ± 0.0695
0.2450 ± 0.0495 0.2243 ± 0.0470
GUT-Q-W
0.2961 ± 0.0206 0.3874 ± 0.0308
0.3462 ± 0.0601
0.3145 ± 0.0539 0.3521 ± 0.0406
Qwen3-1.7B GUT-Q-H
Qwen3-4B
Qwen3-8B
0.2199 ± 0.0227 0.3213 ± 0.0291
0.3297 ± 0.0604
0.2773 ± 0.0594 0.3191 ± 0.0382
GUT-Q-UP 0.3302 ± 0.0238 0.3260 ± 0.0366
0.1531 ± 0.0637
0.3952 ± 0.0512 0.2042 ± 0.0544
GUT-Q-W
0.0764 ± 0.0211 0.1857 ± 0.0307
0.3365 ± 0.0564
0.1531 ± 0.0637 0.2829 ± 0.0464
GUT-Q-H
0.0342 ± 0.0220 0.1581 ± 0.0309
0.3332 ± 0.0623
0.1228 ± 0.0570 0.1259 ± 0.0424
GUT-Q-UP 0.0706 ± 0.0223 0.2121 ± 0.0304
0.2780 ± 0.0558
0.2339 ± 0.0523 0.2767 ± 0.0432
GUT-Q-W
0.1156 ± 0.0208 0.2114 ± 0.0331
0.4276 ± 0.0558
0.1259 ± 0.0557 0.2659 ± 0.0450
GUT-Q-H
0.0887 ± 0.0247 0.1266 ± 0.0327
0.5433 ± 0.0491
0.1128 ± 0.0493 0.2077 ± 0.0361
GUT-Q-UP 0.1452 ± 0.0183 0.2159 ± 0.0327
0.4998 ± 0.0511
0.1684 ± 0.0473 0.3073 ± 0.0366
Table 15: PCCs between the MTLP optimization target and GUT-Q-derived uncertainties across 4 LLMs and 5 datasets.
D. Additional Ablation and Sensitivity Analyses
This appendix provides additional ablation and sensitivity analyses for Qwen3-8B, Qwen3-1.7B, and Qwen3-0.6B, as well as the first-order logic dataset FOLIO [16] and the reasoning QA dataset MMLUPro [45].
D.1. Additional Ablation Analyses Figures 5a, 5b , 5c, and 5d show the ablation comparisons of UQ performance for Qwen3-8B, Qwen3-4B, Qwen3-1.7B, and Qwen3-0.6B, respectively. It is observed that, compared to the blue bars, the red bars are closer while the green bars are notably shorter for the UP methods. This observation demonstrates that node merging is more important than node uncertainty in the UP algorithm. We also observe that the red and green bars are far shorter than the blue bars for both the GUT-Q-W and GUT-Q-H, indicating that both node uncertainty and node merging are essential for width and height calculations.
33
(a) Qwen3-8B
(b) Qwen3-4B
(c) Qwen3-1.7B
(d) Qwen3-0.6B
Figure 5: Ablation comparisons of UQ performance.
34
D.2. Additional Sensitivity Analyses This subsection analyzes how the UQ performance is affected by the number of samples K, temperature T , number of shots F , and node merging criterion for Qwen3-8B, Qwen3-4B, Qwen3-1.7B, and Qwen30.6B. Based on these analyses, we recommend proper configurations of hyperparameters and node merging criteria.
Number of Samples K. Figures 6a, 6b, 6c, and 6d illustrate the impact of the number of samples K on UQ performance for Qwen3-8B, Qwen3-4B, Qwen3-1.7B, and Qwen3-0.6B, respectively. We recommend K = 9 for Qwen3-4B, K = 12 for Qwen3-1.7B, and K = 11 for Qwen3-8B and Qwen3-0.6B to balance efficiency and performance since a larger K leads to higher computational costs.
35
(a) Qwen3-8B
(b) Qwen3-4B
(c) Qwen3-1.7B
(d) Qwen3-0.6B
Figure 6: Impact of the number of samples K on UQ performance.
36
Temperature T . Figures 7a, 7b, 7c, and 7d illustrate the impact of the temperature T on UQ performance for Qwen3-8B, Qwen3-1.7B, and Qwen3-0.6B, respectively. We recommend T = 1.6 for Qwen3-8B, T = 1.0 for Qwen3-4B, and T = 0.7 for both Qwen3-1.7B and Qwen3-0.6B based on the UQ performance.
(a) Qwen3-8B
(b) Qwen3-4B
(c) Qwen3-1.7B
(d) Qwen3-0.6B
Figure 7: Impact of the temperature T on UQ performance.
37
Number of Shots F . Figures 8a, 8b, 8c, and 8d illustrate the impact of the number of shots F on UQ performance for Qwen3-8B, Qwen3-1.7B, and Qwen3-0.6B, respectively. We recommend F = 5 for Qwen3-8B and Qwen3-4B, F = 3 for Qwen3-1.7B, and F = 7 for Qwen3-0.6B based on the UQ performance.
(a) Qwen3-8B
(b) Qwen3-4B
(c) Qwen3-1.7B
(d) Qwen3-0.6B
Figure 8: Impact of the number of shots F on UQ performance.
38
Node Merging Criterion. Following [23], we conducted sensitivity analyses on the criterion for determining equivalence with the NLI model. Specifically, we investigate the following three criteria in Step 15 of Algorithm 5 to determine whether two nodes, denoted as N1 and N2 , are semantically equivalent.
• Bi-entailment. N1 entails N2 and N2 entails N1 , and they do not contradict each other. • Uni-entailment. N1 entails N2 or N2 entails N1 , and they do not contradict each other. • Non-contradiction. N1 and N2 do not contradict each other.
(a) Qwen3-8B
(b) Qwen3-4B
(c) Qwen3-1.7B
(d) Qwen3-0.6B
Figure 9: Impact of the criterion for determining equivalence with the NLI model.
39
Figures 9a, 9b, 9c, and 9d show the impact of the criterion for determining equivalence in the NLI model for Qwen3-8B, Qwen3-4B, Qwen3-1.7B, and Qwen3-0.6B, respectively, where the “bi-entailment” represents the original criterion in Algorithm 5. It is observed that the three criteria yield comparable UQ performance across LLM scales, datasets, and UQ evaluation metrics. Therefore, we can conclude that GUT-Q is robust against the choice of specific criteria of node merging.
E. Case study of the GUT
This appendix presents a case study of instances from MATH-500 on Qwen3-4B to demonstrate our proposed GUT. Figure 10 presents the application of our proposed GUT to two test instances in the MATH-500 dataset with Qwen3-4B, where the left part illustrates an incorrectly answered one and the right part illustrates a correctly answered one. We observe that the graph complexity metrics of the incorrectly answered instance are typically larger than those of the correctly answered one. This observation demonstrates the power of our proposed GUT-Q in discriminating between correct instances and incorrect ones. Moreover, it is obvious that the LLM optimized by GUT-O tends to exhibit fewer reasoning steps for the same question.
Figure 10: Application of our proposed GUT to two test instances from the MATH-500 dataset using Qwen3-4B. For illustrative purposes, the width and height are obtained by removing the node uncertainty in GUT-Q-W and GUT-Q-H, respectively. Now we detail the reasoning space represented by the constructed DAG for these four test instances as follows, where the nodes with boxes denote the final answers, such as 1 .
40
The test instance incorrectly answered by the pretrained Qwen3-4B. √ √ √ √ √ Problem: The number ( 2 + 3)3 can be written in the form a 2 + b 3 + c 6, where a, b, and c are integers. What is a + b + c? Nodes within the constructed DAG: Topological Level=1: √ √ Node 1: Expand ( 2 + 3)3 using the binomial theorem. Topological Level=2: √ √ Node 8: APply the binomial theorem: ( 2 + 3)3 = √ 1 √ 2 √ 0 √ 3 3 3 2 ( 2) ( 3) + 3 ( 2) ( 3) .
√
3 0 (
√ 2)3 ( 3)0 +
√
3 1 (
√ 2)2 ( 3)1 +
Topological Level=3: Node 2: Compute each term of the expansion. Topological Level=4: √ √ √ √ Node 16 : Simplify each term: 2 2 + 3 · 2 · 3 + 3 · 2 · 3 + 3 3. Topological Level=5: √ √ √ √ Node 33 : Calculate the coefficients: 1(2 2) + 3(2 3) + 3(3 2) + 1(3 3). Topological Level=6: √ √ √ √ Node 17 : Combine like terms: 2 2 + 6 3 + 9 2 + 3 3. Topological Level=7: √ √ √ √ √ √ Node 10 : Combine like terms: 2 2 + 9 2 = 11 2, 6 3 + 3 3 = 9 3. Topological Level=8: Node 19 : However, this result is missing the
√
6 term, which suggests that I made a mistake in
the expansion. Topological Level=9: Node 20 : Re-evaluating the binomial expansion, the correct expansion should be √ √ √ √ √ √ ( 2)3 + 3( 2)2 ( 3) + 3( 2)( 3)2 + ( 3)3 . Topological Level=10: √ √ √ √ Node 21 : Correctly simplifying each term: 2 2 + 3 · 2 · 3 + 3 · 2 · 3 + 3 3. Topological Level=11: √ √ √ √ Node 22 : This simplifies to 2 2 + 6 3 + 9 2 + 3 3. Topological Level=12: √ √ √ √ √ Node 11 : Therefore, ( 2 + 3)3 = 11 2 + 9 3 + 0 6. Topological Level=13: √ √ √ Node 3 : Combine like terms to express the result in the form a 2 + b 3 + c 6. √ Node 45 : Recognize that the expression is missing the 6 term, so c = 0.
41
The test instance incorrectly answered by the pretrained Qwen3-4B (Continued). Topological Level=14: Node 4 : Identify the coefficients a, b, and c.
√ √ Node 25 : Rechecking the binomial expansion, the middle terms should be 3( 2)( 3)2 and √ √ √ √ √ √ √ √ 3( 2)2 ( 3), which are 3 2 · 3 = 9 2 and 3 · 2 · 3 = 6 3. However, the cross term 3( 2)( 3) √ should also be included, which is 3 6. Node 12 : The sum a + b + c = 11 + 9 + 0 = 20. Topological Level=15: Node 5 : Add the coefficients a + b + c to get the final answer. √ √ √ √ √ Node 26 : Therefore, the correct expansion is 2 2 + 6 3 + 3 6 + 9 2 + 3 3. Topological Level=16: Node 59 : 4 Node 65 : 2 Node 53 : 3 Node 6 : 1
√ √ √ Node 27 : Combining like terms: 11 2 + 9 3 + 3 6. Topological Level=17: Node 28 : Therefore, a = 11, b = 9, c = 3, and a + b + c = 23. Topological Level=18: Node 29 : 23
The test instance incorrectly answered by the optimized Qwen3-4B. √ √ √ √ √ Problem: The number ( 2 + 3)3 can be written in the form a 2 + b 3 + c 6, where a, b, and c are integers. What is a + b + c? Nodes within the constructed DAG: Topological Level=1: √ √ Node 1 : Expand ( 2 + 3)3 using the binomial theorem. Topological Level=2: Node 2 : The binomial expansion is √ 0 √ 3 3 3 ( 2) ( 3) .
√
3 0 (
√ 2)3 ( 3)0 +
42
√
3 1 (
√ 2)2 ( 3)1 +
√
3 2 (
√ 2)1 ( 3)2 +
The test instance incorrectly answered by the optimized Qwen3-4B (Continued). Topological Level=3: √ √ √ √ √ √ √ Node 3 : Compute each term: 30 ( 2)3 = 1 · 2 2 = 2 2, 31 ( 2)2 ( 3) = 3 · 2 · 3 = 6 3, √ √ √ √ √ √ 2 √ 3 2 · 3 = 9 2, 33 ( 3)3 = 1 · 3 3 = 3 3. 2 ( 2)( 3) = 3 · √ √ √ √ √ √ Node 17 : Simplify each term: ( 2)3 = 2 2, 3( 2)2 ( 3) = 3 × 2 × 3 = 6 3, √ √ √ √ √ √ 3( 2)( 3)2 = 3 × 2 × 3 = 9 2, and ( 3)3 = 3 3. Topological Level=4: √ √ √ Node 26 : Combine like terms to express in the form a 2 + b 3 + c 6. √ √ √ √ √ √ Node 4 : Combine like terms: 2 2 + 9 2 = 11 2, and 6 3 + 3 3 = 9 3. √ √ √ √ Node 61 : Combine like terms: 2 2 + 6 3 + 9 2 + 3 3. Topological Level=5: Node 27 : Identify the coefficients a, b, and c. √ √ √ √ √ Node 5 : Therefore, ( 2 + 3)3 = 11 2 + 9 3 + 0 6. √ Node 19 : However, this does not include the 6 term. This indicates that I might have missed a term in the expansion. Node 62 : Combine the coefficients of
√
2 and
√
√ √ 3: (2 + 9) 2 + (6 + 3) 3.
Topological Level=6: Node 28 : Add the coefficients together to find a + b + c.
√ √ √ √ Node 20 : Rechecking the expansion: the third term should be 3( 2)( 3)2 = 3( 2)(3) = 9 2, √ √ √ and the fourth term is ( 3)3 = 3 3. Therefore, the expansion is correct, and the 6 term is not present.
√ √ Node 63 : Simplify the coefficients: 11 2 + 9 3. Topological Level=7: Node 29 : 1
√ √ √ Node 21 : However, the problem states that the expression can be written as a 2 + b 3 + c 6. √ Since there is no 6 term, it must be that c = 0.
Topological Level=8: Node 6 : The values are a = 11, b = 9, c = 0, so a + b + c = 11 + 9 + 0 = 20. Topological Level=9: Node 13 : Calculate a + b + c = 11 + 9 + 0 = 20. Topological Level=10: Node 7 : 20
43
The test instance correctly answered by the pretrained Qwen3-4B. Problem: For how many two-digit primes is the sum of the digits equal to 8? Nodes within the constructed DAG: Topological Level=1: Node 1 : List all two-digit numbers where the sum of the digits is 8. Topological Level=2: Node 39 : The possible pairs of digits that add up to 8 are (1,7), (2,6), (3,5), (4,4), (5,3), (6,2), (7,1). Topological Level=3: Node 2 : These numbers are 17, 26, 35, 44, 53, 62, 71, and 80. Topological Level=4: Node 3 : Check which of these numbers are prime. Node 20 : The possible two-digit numbers where the sum of the digits is 8 are 17, 26, 35, 44, 53, 62, 71, 80. Topological Level=5: Node 4 : 17, 53, and 71 are prime numbers. Topological Level=6: Node 8 : Count the number of primes found. There are 3 primes. Node 14 : Therefore, the answer is 3. Node 26 : However, 7 is a one-digit prime, so it is not counted. Topological Level=7: Node 5 : 3 Node 27 : Therefore, there are 5 two-digit primes that satisfy the condition. Topological Level=8: Node 28 : 5
44
The test instance correctly answered by the optimized Qwen3-4B. Problem: For how many two-digit primes is the sum of the digits equal to 8? Nodes within the constructed DAG: Topological Level=1: Node 1 : The two-digit primes with digits that sum to 8 are the primes where the sum of the tens and units digits is 8. Node 20 : The two-digit primes with digits adding to 8 are the primes where the digits are (1,7), (2,6), (3,5), (4,4), (5,3), (6,2), (7,1). Topological Level=2: Node 2 : The possible two-digit numbers with digits summing to 8 are 17, 26, 35, 44, 53, 62, 71, 80. Node 21 : The possible two-digit numbers formed from these pairs are 17, 26, 35, 44, 53, 62, 71. Topological Level=3: Node 3 : Among these numbers, the primes are 17, 53, and 71. Topological Level=4: Node 4 : Therefore, there are 3 such two-digit primes. Node 12 : Therefore, the answer is 3. Topological Level=5: Node 5 : 3
References [1] Atf, Z., Safavi-Naini, S. A. A., Lewis, P. R., Mahjoubfar, A., Naderi, N., Savage, T. R., and Soroush, A. (2025). The challenge of uncertainty quantification of large language models in medicine. arXiv preprint arXiv:2504.05278. [2] Bi, Z., Han, K., Liu, C., Tang, Y., and Wang, Y. (2025). Forest-of-thought: Scaling test-time compute for enhancing llm reasoning. In Proceedings of the 42nd International Conference on Machine Learning, pages 4253–4267. [3] Billingsley, P. (1995). Probability and Measure. John Wiley & Sons. [4] Chen, C., Liu, K., Chen, Z., Gu, Y., Wu, Y., Tao, M., Fu, Z., and Ye, J. (2024). INSIDE: LLMs’ internal states retain the power of hallucination detection. In Proceedings of the 12th International Conference on Learning Representations.
45
[5] Chen, Q., Qin, L., Liu, J., Peng, D., Guan, J., Wang, P., Hu, M., Zhou, Y., Gao, T., and Che, W. (2025). Towards reasoning era: A survey of long chain-of-thought for reasoning large language models. arXiv preprint arXiv:2503.09567. [6] Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. (2021). Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. [7] Cormen, T. H., Leiserson, C. E., Rivest, R. L., and Stein, C. (2022). Introduction to Algorithms. MIT press. [8] Da, L., Liu, X., Dai, J., Cheng, L., Wang, Y., and Wei, H. (2025). Understanding the uncertainty of LLM explanations: A perspective based on reasoning topology. In Proceedings of the 2nd Conference on Language Modeling. [9] Darrin, M., Piantanida, P., and Colombo, P. (2023). RainProof: An umbrella to shield text generator from out-of-distribution data. In Proceedings of the 28th Conference on Empirical Methods in Natural Language Processing, pages 5831–5857. [10] Duan, J., Cheng, H., Wang, S., Zavalny, A., Wang, C., Xu, R., Kailkhura, B., and Xu, K. (2024). Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 5050–5063. [11] Fadeeva, E., Vashurin, R., Tsvigun, A., Vazhentsev, A., Petrakov, S., Fedyanin, K., Vasilev, D., Goncharova, E., Panchenko, A., Panov, M., Baldwin, T., and Shelmanov, A. (2023). LM-polygraph: Uncertainty estimation for language models. In Proceedings of the 28th Conference on Empirical Methods in Natural Language Processing, pages 446–461. [12] Farquhar, S., Kossen, J., Kuhn, L., and Gal, Y. (2024). Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625–630. [13] Fomicheva, M., Sun, S., Yankovskaya, L., Blain, F., Guzmán, F., Fishel, M., Aletras, N., Chaudhary, V., and Specia, L. (2020). Unsupervised quality estimation for neural machine translation. Transactions of the Association for Computational Linguistics, 8:539–555. [14] Fu, Y., Wang, X., Tian, Y., and Zhao, J. (2025). Deep think with confidence. arXiv preprint arXiv:2508.15260. [15] Gan, Z., Liao, Y., and Liu, Y. (2025). Rethinking external slow-thinking: From snowball errors to probability of correct reasoning. In Proceedings of the 42nd International Conference on Machine Learning, pages 18170–18188. 46
[16] Han, S., Schoelkopf, H., Zhao, Y., Qi, Z., Riddell, M., Zhou, W., Coady, J., Peng, D., Qiao, Y., Benson, L., Sun, L., Wardle-Solano, A., Szabó, H., Zubova, E., Burtell, M., Fan, J., Liu, Y., Wong, B., Sailor, M., Ni, A., Nan, L., Kasai, J., Yu, T., Zhang, R., Fabbri, A. R., Kryscinski, W., Yavuz, S., Liu, Y., Lin, X. V., Joty, S., Zhou, Y., Xiong, C., Ying, R., Cohan, A., and Radev, D. (2024). FOLIO: natural language reasoning with first-order logic. In Proceedings of the 29th Conference on Empirical Methods in Natural Language Processing, pages 22017–22031. [17] He, P., Liu, X., Gao, J., and Chen, W. (2021). Deberta: Decoding-enhanced bert with disentangled attention. In Proceedings of the 9th International Conference on Learning Representations. [18] Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. (2021). Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874. [19] Horowitz, E., Sahni, S., and Anderson-Freed, S. (1976). Fundamentals of Data Structures, volume 20. Computer Science Press. [20] Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., HatfieldDodds, Z., DasSarma, N., Tran-Johnson, E., et al. (2022). Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221. [21] Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Kaiser, L., Mostafa, S., Pfau, V., Fedus, V., Joo, R., Jain, N., Puri, R., Jiang, A., Alberti, S., Hesse, C., Chen, H., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. (2024). Let’s verify step by step. In Proceedings of the 12th International Conference on Learning Representations. [22] Lin, Z., Trivedi, S., and Sun, J. (2024a). Contextualized sequence likelihood: Enhanced confidence scores for natural language generation. In Proceedings of the 29th Conference on Empirical Methods in Natural Language Processing, pages 10351–10368. [23] Lin, Z., Trivedi, S., and Sun, J. (2024b). Generating with confidence: Uncertainty quantification for black-box large language models. Transactions on Machine Learning Research. [24] Liu, X., Chen, T., Da, L., Chen, C., Lin, Z., and Wei, H. (2025). Uncertainty quantification and confidence calibration in large language models: A survey. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 6107–6117. [25] Malinin, A. and Gales, M. (2021). Uncertainty estimation in autoregressive structured prediction. In Proceedings of the 9th International Conference on Learning Representations. [26] Manakul, P., Liusie, A., and Gales, M. (2023). Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. In Proceedings of the 28th conference on Empirical Methods in Natural Language Processing, pages 9004–9017. 47
[27] Mo, S. and Xin, M. (2024). Tree of uncertain thoughts reasoning for large language models. In Proceedings of the 49th IEEE International Conference on Acoustics, Speech and Signal Processing, pages 12742–12746. [28] Nadeem, M. S. A., Zucker, J.-D., and Hanczar, B. (2009). Accuracy-rejection curves (arcs) for comparing classification methods with a reject option. In Proceedings of the 3rd International Workshop on Machine Learning in Systems Biology, pages 65–81. [29] Nie, Y., Kong, Y., Dong, X., Mulvey, J. M., Poor, H. V., Wen, Q., and Zohren, S. (2024). A survey of large language models for financial applications: Progress, prospects and challenges. arXiv preprint arXiv:2406.11903. [30] Nikitin, A., Kossen, J., Gal, Y., and Marttinen, P. (2024). Kernel language entropy: Fine-grained uncertainty quantification for llms from semantic similarities. Advances in Neural Information Processing Systems 37, pages 8901–8929. [31] Prabhudesai, M., Chen, L., Ippoliti, A., Fragkiadaki, K., Liu, H., and Pathak, D. (2025). Maximizing confidence alone improves reasoning. arXiv preprint arXiv:2505.22660. [32] Qiu, X. and Miikkulainen, R. (2024). Semantic density: Uncertainty quantification for large language models through confidence measurement in semantic space. In Advances in Neural Information Processing Systems 37, pages 134507–134533. [33] Raiffa, H. (1968). Decision Analysis: Introductory Lectures on Choices Under Uncertainty. AddisonWesley. [34] Ren, J., Luo, J., Zhao, Y., Krishna, K., Saleh, M., Lakshminarayanan, B., and Liu, P. J. (2023). Out-of-distribution detection and selective generation for conditional language models. In Proceedings of the 11th International Conference on Learning Representations. [35] Scarselli, F., Gori, M., Tsoi, A. C., Hagenbuchner, M., and Monfardini, G. (2008). The graph neural network model. IEEE Transactions on Neural Networks, 20(1):61–80. [36] Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., et al. (2024). Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. [37] Sriramanan, G., Bharti, S., Sadasivan, V. S., Saha, S., Kattakinda, P., and Feizi, S. (2024). Llm-check: Investigating detection of hallucinations in large language models. In Advances in Neural Information Processing Systems 37, pages 34188–34216.
48
[38] Takayama, J. and Arase, Y. (2019). Relevant and informative response generation using pointwise mutual information. In Proceedings of the 1st Workshop on NLP for Conversational AI, pages 133–138. [39] van der Poel, L., Cotterell, R., and Meister, C. (2022). Mutual information alleviates hallucinations in abstractive summarization. In Proceedings of the 27th Conference on Empirical Methods in Natural Language Processing, pages 5956–5965. [40] Vashurin, R., Fadeeva, E., Vazhentsev, A., Rvanova, L., Vasilev, D., Tsvigun, A., Petrakov, S., Xing, R., Sadallah, A., Grishchenkov, K., et al. (2025a). Benchmarking uncertainty quantification methods for large language models with lm-polygraph. Transactions of the Association for Computational Linguistics, 13:220–248. [41] Vashurin, R., Goloburda, M., Ilina, A., Rubashevskii, A., Nakov, P., Shelmanov, A., and Panov, M. (2025b). Uncertainty quantification for llms through minimum bayes risk: Bridging confidence and consistency. arXiv preprint arXiv:2502.04964. [42] Vazhentsev, A., Rvanova, L., Kuzmin, G., Fadeeva, E., Lazichny, I., Panchenko, A., Panov, M., Baldwin, T., Sachan, M., Nakov, P., et al. (2025). Uncertainty-aware attention heads: Efficient unsupervised uncertainty quantification for llms. arXiv preprint arXiv:2505.20045. [43] Wang, X. and Zhou, D. (2024). Chain-of-thought reasoning without prompting. Advances in Neural Information Processing Systems 37, pages 66383–66409. [44] Wang, Y., Jiao, R., Zhan, S. S., Lang, C., Huang, C., Wang, Z., Yang, Z., and Zhu, Q. (2023). Empowering autonomous driving with large language models: A safety perspective. arXiv preprint arXiv:2312.00812. [45] Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., et al. (2024). Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In Advances in Neural Information Processing Systems 37, pages 95266–95290. [46] Williams, A., Nangia, N., and Bowman, S. (2018). A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 16th Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1112–1122. [47] Xiong, M., Hu, Z., Lu, X., LI, Y., Fu, J., He, J., and Hooi, B. (2024). Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. In Proceedings of the 12th International Conference on Learning Representations.
49
[48] Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., Zheng, C., Liu, D., Zhou, F., Huang, F., Hu, F., Ge, H., Wei, H., Lin, H., Tang, J., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., and Zhou, J. (2025). Qwen3 technical report. arXiv preprint arXiv:2505.09388. [49] Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., and Narasimhan, K. (2023). Tree of thoughts: Deliberate problem solving with large language models. In Advances in Neural Information Processing Systems 36, pages 11809–11822. [50] Ye, F., Yang, M., Pang, J., Wang, L., Wong, D., Yilmaz, E., Shi, S., and Tu, Z. (2024). Benchmarking llms via uncertainty quantification. In Advances in Neural Information Processing Systems 37, pages 15356–15385. [51] Yin, Z., Sun, Q., Guo, Q., Zeng, Z., Li, X., Dai, J., Cheng, Q., Huang, X.-J., and Qiu, X. (2024). Reasoning in flux: Enhancing large language models reasoning through uncertainty-aware adaptive guidance. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 2401–2416. [52] Zhang, B. and Zhang, R. (2025). CoT-UQ: Improving response-wise uncertainty quantification in llms with chain-of-thought. arXiv preprint arXiv:2502.17214. [53] Zhang, C., Liu, F., Basaldella, M., and Collier, N. (2024a). LUQ: Long-text uncertainty quantification for LLMs. In Proceedings of the 28th Conference on Empirical Methods in Natural Language Processing, pages 5244–5262. [54] Zhang, C., Shu, C., Shareghi, E., and Collier, N. (2025). All roads lead to rome: Graph-based confidence estimation for large language model reasoning. In Proceedings of the 30th Conference on Empirical Methods in Natural Language Processing, pages 31802–31812. [55] Zhang, M., Press, O., Merrill, W., Liu, A., and Smith, N. A. (2024b). How language model hallucinations can snowball. In Proceedings of the 41st International Conference on Machine Learning, pages 59670–59684. [56] Zhao, X., Kang, Z., Feng, A., Levine, S., and Song, D. (2025). Learning to reason without external rewards. arXiv preprint arXiv:2505.19590. [57] Zhu, Y., Li, G., Jiang, X., Li, J., Mei, H., Jin, Z., and Dong, Y. (2025). Uncertainty-guided chain-of-thought for code generation with llms. arXiv preprint arXiv:2503.15341.
50