Using Semantic Distance to Estimate Uncertainty in LLM-Based Code Generation
arXiv:2605.09023v1 [cs.SE] 9 May 2026
Weilin He University of Bristol [email protected]
Arindam Sharma University of Bristol [email protected]
Cristina David University of Bristol [email protected]
Abstract LLMs show strong performance in code generation, but their outputs lack correctness guarantees. Sample-based uncertainty estimators address this by generating multiple candidate programs and measuring their disagreement. However, existing estimators make different design choices about how behaviours are identified, aggregated, referenced and compared, making them difficult to assess. We therefore first introduce a taxonomy that disentangles these choices and reveals a missing design point: semantic distance-aware uncertainty estimation, which measures not only whether sampled programs disagree, but how severely their execution behaviours differ. Across LiveCodeBench, MBPP, HumanEval-X and BigCodeBench, spanning Python, Java and C++, our metrics provide strong proxies for correctness, and consistently outperform state-of-the-art sample-based baselines across both closed-source models (GPT-3.5-Turbo, GPT-4o-mini, Gemini-2.5-Flash-Lite, Claude Opus 4.5) and an open-source model (DeepSeek-Coder-V2). The method is practical: it requires neither model internals nor LLM-as-judge calls, remains robust across models, languages, sampling temperatures and fuzzing settings, and reduces runtime by approximately 48–79% relative to existing baselines.
1
Introduction
The adoption of Large Language Models (LLMs) in software engineering has been rapid, with the volume of LLM-generated code projected to increase further [6]. As such, the well-known problem of hallucinations now threatens the reliability of critical software systems [19], spurring a growing body of work on assessing the correctness of LLM-generated code [9]. Yet, in many settings, external validation is unavailable or incomplete: generated programs may lack reference implementations or complete test suites [13, 21, 22]. A common reference-free strategy is to estimate uncertainty [14] from disagreement across multiple independently sampled outputs. In natural language generation, such methods interpret variation across samples as a signal of epistemic uncertainty [7, 16, 1]. Recent work adapts the same principle to code generation [17, 23, 24], using signals that range from static analysis of program text to dynamic comparison of execution behaviour. A four-axis taxonomy of sample-based uncertainty estimators. Although many uncertainty estimators now exist, they are difficult to compare directly: they combine different notions of equivalence, aggregation, reference behaviour and disagreement. Our first contribution is therefore a taxonomy that makes these choices explicit. Preprint.
Table 1: Representative sample-based uncertainty estimators positioned along four design axes. Equiv. proxy denotes how outputs are represented or grouped into behaviours. Aggregation is marginal or pairwise. Reference specifies which behaviours are compared for pairwise methods. Dissimilarity specifies how compared behaviours are scored; N/A indicates that no pairwise comparison is used. Method
Equiv. proxy
Aggregation
Reference
Dissimilarity
Semantic Entropy [7, 16] Symbolic equiv. [23] Self-Consistency [25] SelfCheckGPT [20] ChainPoll [8] HonestCoder [17] DiffTrust [24] EigenScore [18]
NL-Entailment symbolic execution exact string match NL-LLM LLM voting multi-modal heuristic execution fuzz NL-Embedding
marginal marginal marginal pairwise pairwise pairwise pairwise pairwise
N/A N/A N/A uniform-pairwise top1-only uniform-pairwise top-anchored uniform-pairwise
N/A N/A N/A binary binary binary binary binary
SDE (ours) DSDE (ours)
execution fuzz execution fuzz
pairwise pairwise
uniform-pairwise top-anchored
graded graded
At a high level, the methods in Table 1 share a common sample-and-aggregate structure: they sample K candidate outputs, identify or compare behaviours using some proxy for equivalence, and aggregate the resulting evidence into a scalar uncertainty score. We organise this design space along four axes: equivalence proxy, aggregation structure, reference behaviour and dissimilarity. Equivalence proxy specifies how candidate outputs are represented or grouped into behaviours. Existing methods use a range of proxies, including NLI- or LLM-based equivalence judgements [20], syntactic and data-flow heuristics [17], symbolic execution [23], execution-based grouping on shared inputs [24], exact string matching [25], and embedding similarity [18]. Aggregation structure specifies whether uncertainty is computed from marginal behaviour probabilities or from pairwise comparisons between behaviours. Marginal methods [7, 23, 25] summarise how probability mass is spread across behaviours, without asking whether two behaviours are semantically close or far apart. Pairwise methods [20, 8, 17, 24, 18], by contrast, explicitly compare behaviours. Reference behaviour applies to pairwise methods and specifies which behaviours are compared. Some methods aggregate over all pairs (uniform-pairwise) [20, 17, 18]; some compare the dominant behaviour against alternatives (top-anchored) [24]; and some compare only the top-ranked output against alternatives while discarding alternative probabilities (top1-only) [8]. Dissimilarity applies to pairwise methods and specifies how differences between compared behaviours are scored. Existing pairwise methods use a binary or shallow notion of disagreement: two behaviours either agree or disagree [20, 8, 24]. Marginal methods, such as Semantic Entropy [7], do not use an inter-behaviour dissimilarity at all. Limitation of existing methods. By separating design choices, the taxonomy makes prior methods easier to compare and exposes a key limitation: existing estimators provide only a coarse account of behavioural difference. Marginal methods collapse uncertainty to the distribution of mass over behaviours, without asking whether those behaviours are semantically close or far apart. Pairwise methods do compare behaviours, but typically only through a binary or shallow agree/disagree signal. As a result, two programs that differ on one input out of ten may be treated like two programs that disagree on every input. This is a poor abstraction for code, where behavioural differences are often graded [27]. A useful sample-based uncertainty estimator should therefore capture not only whether sampled programs disagree, but also how severely they disagree. Our approach. Our key observation is that programs, unlike free-form natural language, are executable objects. This makes semantic difference directly measurable: two programs can be compared by how often, and in what ways, their executions differ on shared inputs. We therefore propose semantic distance-aware uncertainty estimation. Instead of treating all behavioural disagreements as equally severe, we assign graded dissimilarities between execution behaviours and aggregate them into uncertainty scores. The framework is agnostic to the equivalence proxy: any procedure that identifies behaviourally equivalent programs can be used. In this paper, we 2
use fuzzing: unlike lightweight static proxies [17], it captures observable program behaviour, while avoiding the substantial cost of more principled static techniques such as symbolic execution [23]. We define two metrics, corresponding to two reference behaviours. SDE uses uniform-pairwise aggregation, grounded in Rao’s quadratic entropy, and measures global behavioural diversity across sampled programs. DSDE uses top-anchored aggregation, measuring how strongly the alternatives disagree with the dominant behaviour containing the top-ranked program. The former is suited to task- or model-level uncertainty, while the latter targets the reliability of the specific output shown to the user. In summary, the contributions of this paper are as follows: • We introduce a four-axis taxonomy of sample-based uncertainty estimators, making existing methods easier to compare and clarifying which design choices distinguish them. • We propose semantic distance-aware uncertainty metrics as a proxy for correctness in code generation. On LiveCodeBench, our metrics achieve strong discrimination between correct and incorrect solutions, with AUROC > 0.8. • We provide extensive evidence of generalisation across benchmarks, languages, models, and sampling regimes. Beyond LiveCodeBench, our metrics remain effective on MBPP [2], BigCodeBench [29] and HumanEval-X [28]; across Python, Java and C++; across closed-source and open-weights models; and under varying task difficulties and sampling temperatures. • We show that distance-aware uncertainty is cost-effective relative to strong state-of-the-art uncertainty estimation baselines: our metrics substantially improve predictive performance while reducing runtime by approximately 48–79%.
2
Distance-Aware Uncertainty Measures
We now formalise the three components needed to compute our distance-aware uncertainty scores: semantic clusters, distances between clusters, and the aggregation rules that turn these distances into a scalar uncertainty estimate. The clustering step is not specific to our metric: any procedure that partitions candidate programs into behaviourally equivalent groups can be used. In this paper, we use execution signatures over a shared input set, which provides a simple and practical instantiation. Setup: Semantic Clusters. Given a task description d, let Π = {π1 , . . . , πK } be K candidate programs sampled from a stochastic code-generation model. Let I = (x1 , . . . , xN ) be a fixed set of inputs shared across all candidates. Executing a program π on I yields its execution signature σI (π) = o1 (π), . . . , oN (π) , where ok (π) is the observed outcome of π on input xk . An outcome is either a normal output value or an abnormal termination labelled by its error type. Programs are placed in the same semantic cluster iff their execution signatures are identical. Let {C1 , . . . , CM } be the resulting partition of Π. The empirical cluster probability is pi = |CKi | , with PM i=1 pi = 1. 2.1
Semantic Distance
We next define a graded distance between semantic clusters. Since all programs in a cluster have the same execution signature on I, the distance between two clusters is independent of the choice of representatives. For any π ∈ Ci and π ′ ∈ Cj , the semantic distance between clusters Ci and Cj is dij =
N 1 X δ ok (π), ok (π ′ ) . N
(1)
k=1
The per-input outcome distance δ(o, o′ ) ∈ [0, 1] is defined as 0 if o and o′ are equal normal outputs, ′ 1 if both outcomes are normal and o ̸= o , ′ δ(o, o ) = a if exactly one outcome is abnormal, b if both outcomes are abnormal with different error types, c if both outcomes are abnormal with the same error type. 3
(2)
where a, b, c ∈ [0, 1] are graded disagreement costs. We treat these values as hyperparameters because abnormal executions do not induce a canonical semantic distance: identical errors may indicate related failure modes, but need not imply behavioural equivalence. We use a fixed default in the main experiments (see §4.1) and assess sensitivity in §4.5. By construction, dij is symmetric, satisfies dii = 0, and lies in [0, 1]. 2.2
Uncertainty Metrics
Given semantic clusters {C1 , . . . , CM }, empirical probabilities {p1 , . . . , pM }, and pairwise distances {dij }, we define two distance-aware uncertainty metrics. They share the same clustering and distance definitions, but differ in their anchor: one aggregates globally over all cluster pairs, while the other anchors uncertainty to the cluster containing the top-ranked output. Semantic Distance Entropy (SDE). Our first measure is a symmetric distance-aware estimator inspired by Rao’s quadratic entropy, weighting each cluster pair by the product of cluster probabilities and their semantic distance: X SDE = pi pj dij . (3) i<j
SDE captures global behavioural diversity: it is large when substantial probability mass is spread across clusters that are far apart in execution behaviour. Dominant Semantic Distance Entropy (DSDE). In deployment, users typically inspect or rely on the model’s top-ranked output [5]. We therefore define a top-anchored variant that measures uncertainty relative to the behaviour that would actually be served. Let c∗ ∈ {1, . . . , M } be the index of the cluster containing the top-ranked program π1 , so that Cc∗ = C(π1 ). We define X DSDE = pi dc∗ ,i . (4) i̸=c∗
DSDE measures how strongly the alternatives disagree with the top-ranked behaviour, weighted by the empirical probability of each alternative. Unlike SDE, which treats all cluster pairs symmetrically, DSDE is targeted at the reliability of the specific output shown to the user.
3
Overview of the Approach
We present an end-to-end evaluation pipeline for execution-based uncertainty estimation in code generation, which consists of four main stages: First stage: Candidate program sampling. Given a programming task described by natural language input d, let Coder(·) denote a code generation system. Without access to reference implementations or model internals, the system independently samples K candidate programs {π1 , π2 , . . . , πK }, πk ∼ p(π | d), where each πk corresponds to the raw code returned by API call. Second stage: Fuzzing-based input generation. Starting from an initial set of valid task inputs, we generate fuzzed inputs using a lightweight, type-aware mutation strategy designed for the heterogeneous input spaces common in code-generation benchmarks. We apply structure-preserving, type-specific mutations iteratively until N executable fuzzed inputs are obtained, providing behavioural variation for execution-based equivalence and uncertainty estimation. By default, we follow Valentin et al. [24] and use benchmark-provided test cases as seed inputs. However, our pipeline does not rely on such seeds: in §4.5, we show that a seed-free variant remains effective. We also report input-quality diagnostics for the resulting fuzz suites, including validity, uniqueness, and coverage, in Appendix A. More generally, the pipeline is fuzzer-agnostic: the input-generation component can be replaced by any suitable fuzzing strategy, allowing us to inherit advances from the fuzzing community. For example, in §4.3, we use whitebox fuzzing [10] for BigCodeBench. Third stage: Execution-based representation and clustering. Let i = (i1 , i2 , . . . , iN ) denote the set of test inputs associated with the task, where N is the number of inputs. This input set is shared 4
across all candidate programs π = (π1 , π2 , . . . , πK ). Each candidate program πk is executed on all inputs in i, yielding an execution output vector ok = πk (i1 ), πk (i2 ), . . . , πk (iN ) . Candidate programs are clustered according to behavioural equivalence: programs with identical execution signatures are assigned to the same behavioural cluster. For abnormal executions, programs are grouped into the same cluster if they terminate with the same error. Fourth stage: Uncertainty computation. Based on the semantic clusters, we compute SDE and DSDE.
4
Experiments
In this section, we will evaluate our distance-aware uncertainty metrics. Our experiments are designed to answer the following questions: RQ1 (Predictive Performance). Do distance-aware uncertainty metrics reliably estimate program correctness, and how do they compare to existing baselines [24, 17, 7]? RQ2 (Generalisation). Do distance-aware uncertainty metrics transfer across benchmarks of varying difficulty and scale, programming languages, sampling temperatures and model families? RQ3 (Cost-effectiveness). Are distance-aware uncertainty metrics cheap enough for inference-time deployment under aggressive sample budgets and competitive against runtime baselines? RQ4 (Robustness to design choices). Do distance-aware uncertainty metrics remain reliable under different design choices, including fixed versus tuned distance weights and the absence of benchmarkprovided test inputs?
4.1
Experimental Setup
Models. We conduct experiments using representative code generation models spanning different capability tiers: GPT-3.5-Turbo (low), GPT-4o-mini (medium), Gemini-2.5-Flash-Lite (medium), and Claude Opus 4.5 (high), with respective pass@1 success rates of 30.83%, 49.24%, 60.53%, and 82.73% on LiveCodeBench. To assess generalisation (§4.3), we additionally use the open-weights model DeepSeek-Coder-V2 (pass@1 38.14% on LiveCodeBench). Datasets. We evaluate on four complementary benchmarks spanning different task difficulties and programming languages. Our primary benchmark is LiveCodeBench (LCB) [15] (Python). To assess generalisation (§4.3), we additionally use MBPP [2] (Python), HumanEval-X [28] (Python, Java, and C++), and BigCodeBench [29] (Python). Detailed statistics and task descriptions are in Appendix C.3. Experimental configuration. Unless otherwise specified, we sample with temperature T = 0.6 following prior code-generation work [17, 11], generate K = 10 candidate programs per task, and evaluate them on N = 10 fuzz inputs with a 0.2 s per-input execution timeout. We use (a, b, c) = (1, 0.8, 0.6) from §2.1; a sensitivity analysis shows the fixed weights deviate by less than 0.011 AUROC from per-setting learned weights, more details can be found in §4.4. All experiments are run on an Apple M4 Pro with 24 GB of RAM. Prediction targets. Following prior work [5], we adopt pass@1 as a primary correctness signal, capturing whether the model succeeds on its first attempt. Since pass@1 is binary, it does not distinguish near-correct programs from substantially incorrect ones. We therefore use partial_pass@1 as a complementary, graded execution-based target, defined as the proportion of test cases passed by the first sampled program, as in [23]. This provides a useful measure of partial correctness. Evaluation metrics. For pass@1, which is a binary outcome, we evaluate how well uncertainty scores discriminate between correct and incorrect generations using the area under the ROC curve (AUROC) [4]. For the continuous-valued partial_pass@1, we assess its association with uncertainty scores using Pearson [3] and Spearman [26] correlation coefficients, capturing linear and monotonic relationships, respectively. 5
4.2
Answering RQ1 (Predictive Performance)
pass@1 and partial_pass@1 prediction. Table 2 reports AUROC for pass@1 failure prediction and Pearson r/Spearman ρ correlations with partial_pass@1 on LiveCodeBench. Across all four models, SDE and DSDE provide strong predictive signals: DSDE achieves AUROC above 0.8 in every setting, and both metrics correlate negatively with partial_pass@1, indicating that higher uncertainty corresponds to lower partial correctness. On the main closed-source LiveCodeBench evaluation, DSDE improves over SDE across all models and metrics, supporting the top-anchored design: when assessing the program shown to the user, uncertainty can be more informative when measured relative to that program’s behaviour rather than averaged uniformly across all samples. This advantage may depend on the first sample being representative; when it is not, as suggested by DeepSeek-Coder-V2 in Table 3, the symmetric SDE estimator can be more stable. Comparison with baselines. We compare against three strong baselines, chosen to cover the main existing design choices in Table 1: DiffTrust, the state-of-the-art incoherence-based proxy [24]; HonestCoder, a representative confidence estimator combining embeddings, data-flow signatures, and syntactic n-grams [17]; and Semantic Entropy, the standard entropy-based uncertainty estimator over LLM-judged semantic clusters [7]. As shown in Table 2, even these competitive baselines are only weakly informative on LiveCodeBench. By contrast, SDE and DSDE outperform every baseline on every model and metric, suggesting that graded semantic distance captures correctness-relevant information missed by binary disagreement and marginal entropy-based uncertainty. Table 2: Performance of SDE, DSDE, and the three baselines on LiveCodeBench across models (T = 0.6). Best result per metric per row is shown in bold. AUC denotes AUROC; r and ρ are Pearson and Spearman correlations with partial_pass@1, respectively. Per-difficulty breakdowns (Easy/Medium/Hard) are reported in Appendix C.1. SDE
DSDE
DiffTrust
HC
Semantic Entropy
Model
AUC
r
ρ
AUC
r
ρ
AUC
r
ρ
AUC
r
ρ
AUC
r
ρ
GPT-3.5-Turbo-0125 GPT-4o-mini Gemini-2.5-Flash-Lite Claude-opus-4.5
0.827 0.826 0.794 0.818
−0.587 −0.589 −0.494 −0.552
−0.581 −0.584 −0.491 −0.450
0.844 0.844 0.808 0.825
−0.622 −0.624 −0.532 −0.611
−0.620 −0.624 −0.512 −0.461
0.669 0.534 0.688 0.606
−0.259 −0.024 −0.296 −0.139
−0.267 −0.064 −0.298 −0.139
0.646 0.646 0.668 0.739
−0.344 −0.344 −0.301 −0.424
−0.328 −0.328 −0.329 −0.350
0.605 0.607 0.639 0.687
−0.240 −0.240 −0.237 −0.275
−0.237 −0.238 −0.242 −0.287
4.3
Answering RQ2 (Generalisation)
Across benchmarks of varying difficulty. The first block of Table 3 evaluates SDE and DSDE on three Python benchmarks spanning entry-level function completion (MBPP), competitive programming (LiveCodeBench), and repository-scale tasks (BigCodeBench). Both metrics retain strong predictive performance on MBPP and LiveCodeBench, with AUROC above 0.75 and Spearman correlations close to −0.6. Beyond function-level benchmarks: BigCodeBench. BigCodeBench is the most demanding benchmark we consider and, to our knowledge, the first repository-scale evaluation of sample-based uncertainty estimation for LLM-generated code. Its tasks include cross-file dependencies, side effects, stochastic operations, and often a range of valid outputs for the same specification. This makes exact-match execution clustering too brittle: semantically valid programs may produce different concrete outputs and be incorrectly separated. Our formulation can accommodate this setting because it is agnostic with respect to the clustering proxy. We therefore make a single drop-in change: instead of clustering by exact output match, we use whitebox fuzzing [10] to target specification predicates and group programs by predicate satisfaction. With this adaptation, DSDE achieves AUROC 0.668 and Spearman ρ = −0.335 on BigCodeBench. Although lower than on function-level benchmarks, this remains a non-trivial signal in a substantially more challenging repository-scale regime. Across programming languages. On HumanEval-X [28], covering Python, Java, and C++, both metrics perform consistently across languages without language-specific tuning. 6
Across sampling temperatures. We sweep the sampling temperature T ∈ {0.2, 0.6, 0.8} using GPT-4o-mini on LiveCodeBench. As shown in Table 3, performance improves with temperature: lower temperatures reduce sample diversity, while higher temperatures expose richer behavioural variation for uncertainty estimation. Nevertheless, SDE and DSDE remain informative across all settings, indicating that distance-aware uncertainty benefits from, but does not depend on, high sampling diversity. Across model families. Finally, we evaluate on the open-weights DeepSeek-Coder-V2 model, which achieves pass@1 of 38.14% on LiveCodeBench. Both metrics retain meaningful AUROC and correlations with partial_pass@1, showing that the distance-aware signal transfers beyond the closed-source models used in RQ1. Table 3: Performance of SDE and DSDE across benchmarks, languages, sampling temperatures, and model families (GPT-4o-mini, T = 0.6, unless noted). Better of SDE/DSDE per row in AUROC in bold. Per-difficulty LiveCodeBench breakdowns are in Appendix C.1. Pearson r
AUROC Benchmark
SDE
DSDE
Spearman ρ
SDE
DSDE
SDE
DSDE
0.752 0.844 0.668
−0.480 −0.589 −0.281
−0.434 −0.624 −0.287
−0.601 −0.584 −0.333
−0.595 −0.624 −0.335
0.757 0.745 0.804
−0.523 −0.509 −0.599
−0.521 −0.516 −0.619
−0.631 −0.605 −0.573
−0.634 −0.603 −0.587
0.780 0.826 0.837
0.779 0.844 0.853
−0.538 −0.589 −0.612
−0.529 −0.624 −0.660
−0.524 −0.584 −0.609
−0.522 −0.624 −0.646
0.778
0.769
−0.483
−0.469
−0.479
−0.462
Increased difficulty (Python benchmarks, easy to hard) MBPP LiveCodeBench BigCodeBench
0.751 0.826 0.664
Programming languages (HumanEval-X) Python Java C++
0.751 0.740 0.796
Sampling temperature (GPT-4o-mini, LCB) T = 0.2 T = 0.6 T = 0.8 Open-weights model (LCB) DeepSeek-Coder-V2
4.4
Answering RQ3 (Cost-effectiveness)
Scaling K and N up to test for further gains. To check whether allocating a larger budget improves predictive performance, we extend the sensitivity analysis on a 50-task LiveCodeBench subset across all four code-generation models. Sweeping K ∈ {5, 10, 15, 20} at fixed N = 10 leaves DSDE AUROC within 0.024 of the default (K, N ) = (10, 10) across all values; sweeping N from 10 up to 1,000 at fixed K = 10 likewise produces no measurable trend (Figure 1). Both sweeps indicate that scaling K or N beyond the default (K, N ) = (10, 10) yields no measurable predictive gain. Claude 0.35
GPT-3.5
GPT-4o
0.40
0.76 0.74
0.45 0.50 0.55
0.72 10 20
50 100
500 1000
Number of fuzz inputs N
0.35 0.40
Spearman
0.78
Pearson r
AUROC
0.80
Gemini
0.45 0.50 0.55
10 20
50 100
500 1000
Number of fuzz inputs N
10 20
50 100
500 1000
Number of fuzz inputs N
Figure 1: Sensitivity of DSDE to the number of fuzz inputs N on a 50-task LiveCodeBench subset, with K fixed to 10. AUROC, Pearson r, and Spearman ρ remain essentially flat across N ∈ {10, 20, 50, 100, 500, 1000} for all four code-generation models. Scaling K and N down to find the minimal viable configuration. Since enlarging the budget does not help, we instead probe how far (K, N ) can be reduced before the predictive signal degrades. 7
We sweep K, N ∈ {3, 5, 8, 10} jointly (Figure 2). The smallest configuration (K, N ) = (3, 3) already yields AUROC 0.783, recovering ∼93% of the AUROC at (10, 10) while requiring an orderof-magnitude less sample and execution cost. Therefore, (3, 3) can be adopted as a cost-effective deployment default for budget-constrained settings.
N=10 0.8081 0.8321 0.8400 0.8442 K=3
K=5
K=8
0.607
N=5 0.5504 0.5858 0.6067 0.6085
0.774
0.575
N=8 0.5637 0.5982 0.6125 0.6205 0.543
N=10 0.5665 0.6033 0.6173 0.6244
K=10
K=3
Number of Generated Programs (K)
K=5
K=8
0.511
K=10
Number of Generated Programs (K)
0.639
N=3 0.5215 0.5597 0.5808 0.5879
0.606
N=5 0.5455 0.5828 0.6049 0.6077 0.573
Value
0.794
0.639
Value
0.814
N=8 0.8068 0.8290 0.8385 0.8420
N=3 0.5261 0.5623 0.5835 0.5896
Number of Fuzz Inputs (N)
0.833
N=5 0.8005 0.8230 0.8342 0.8365
Number of Fuzz Inputs (N)
N=3 0.7830 0.8093 0.8228 0.8265
Pearson Correlation (Partial Pass@1) - DSDE Spearman Correlation (Partial Pass@1) - DSDE 0.853
Value
Number of Fuzz Inputs (N)
AUROC (Pass@1) - DSDE
N=8 0.5614 0.5970 0.6113 0.6197 0.539
N=10 0.5621 0.6014 0.6159 0.6241 K=3
K=5
K=8
0.506
K=10
Number of Generated Programs (K)
Figure 2: Joint cost-effectiveness analysis of DSDE on LiveCodeBench (GPT-4o-mini, T = 0.6), varying K ∈ {3, 5, 8, 10} and N ∈ {3, 5, 8, 10}. Runtime against baselines. We now compare end-to-end wall-clock cost against other uncertainty estimators. Table 4 breaks runtime into: candidate-program sampling, additional model-side computation (Model Cost, e.g. embeddings or LLM-as-judge calls), fuzz-input generation, program execution and metric computation. The sampling cost is shared across methods; the main differences arise from the additional model-side or execution costs required by each estimator. At the default (K, N ) = (10, 10), our method takes on average ∼5.7 s per task, compared with ∼11 s for HonestCoder, ∼13 s for Semantic Entropy, and ∼27 s for DiffTrust. This is a ∼79% reduction relative to state-of-the-art DiffTrust. With the deployment default (K, N ) = (3, 3), the runtime falls further to ∼2 s per task. Table 4: Per-task runtime breakdown on LiveCodeBench with GPT-4o-mini. Method
N
Sampling
Model Cost
Fuzzing
Execution
Metric
Total
HonestCoder Semantic Entropy DiffTrust
— — 1000
∼3 s ∼3 s ∼3 s
∼7 s ∼10 s —
— — ∼3 s
— — ∼20 s
∼1 s <1 ms ∼1 s
∼11 s ∼13 s ∼27 s
Ours
10
∼3 s
—
<5 ms
∼2.7 s
<1 ms
∼5.7 s
The saving comes from two design choices: (i) distance-aware uncertainty metrics avoid model costs entirely (no embeddings, no LLM-as-judge); and (ii) while DiffTrust requires N = 1,000 test inputs by default to satisfy the variance bounds of its Monte Carlo incoherence estimator, our metric is a deterministic pairwise aggregate over execution clusters and does not depend on a large N for statistical guarantees. 4.5
Answering RQ4 (Robustness to Design Choices)
Robustness to graded disagreement costs (a, b, c). The graded disagreement costs (a, b, c) are fixed to (1, 0.8, 0.6) throughout the experiments above. To quantify the gap between this fixed scheme and a per-setting tuned alternative, we additionally fit (a, b, c) via an 80:20 train–test split per model on LiveCodeBench at T = 0.6. As shown in Table 5, the default and learned schemes differ by less than 0.011 AUROC across all four models, indicating that the fixed weights are sufficient on LiveCodeBench. In our current experiments, the default setting appears to be stable and strong, but learned weights may still be useful in future settings or datasets with different failure patterns. Robustness under seed-free input generation (no dataset-provided test inputs). In contrast to the seeded experiments above (where the fuzzer mutates benchmark-provided test inputs as initial seeds), we further evaluate a seed-free variant in which the fuzzer never reads benchmark-provided test cases, expected outputs, or reference implementations. Parameter types are inferred by a rule-based parser from the function interface, using type annotations when available and parameter names or natural-language hints from the prompt otherwise. Inputs are then sampled from generic type-level 8
Table 5: Robustness to distance weights (a, b, c): fixed default (1.0, 0.8, 0.6) vs. per-setting learned weights on LiveCodeBench (GPT-4o-mini, T = 0.6, 80:20 train–test split per model). (a, b, c)
Default
Learned
∆
(0.90, 0.90, 0.25) (0.90, 0.90, 0.90) (0.97, 0.88, 0.88) (1.00, 0.80, 0.80)
0.8248 0.8075 0.8442 0.8442
0.8206 0.7974 0.8395 0.8524
−0.0042 −0.0101 −0.0047 +0.0082
Setting Claude Opus 4.5 Gemini-2.5-Flash-Lite GPT-3.5-Turbo GPT-4o-mini
distributions. As shown in Table 6, the AUROC degradation is small (at most 0.087) and the predictive signal remains substantially above chance across all benchmarks, confirming that SDE and DSDE are fuzzer-agnostic. Table 6: Robustness to fuzz-input seeds: AUROC, Pearson, and Spearman across three fuzzer seeds on LiveCodeBench (GPT-4o-mini, T = 0.6). SDE
DSDE
Benchmark
Seed
Seed-free
∆
Seed
Seed-free
∆
MBPP LiveCodeBench HumanEval-X (Python) HumanEval-X (Java) HumanEval-X (C++)
0.751 0.826 0.751 0.748 0.796
0.712 0.786 0.705 0.725 0.709
−0.039 −0.040 −0.046 −0.023 −0.087
0.752 0.844 0.757 0.745 0.804
0.711 0.787 0.704 0.731 0.723
−0.041 −0.057 −0.053 −0.014 −0.081
Robustness to fuzz-input random seeds. Here, “seed” refers to the random seed of the fuzz-input generator, not to the benchmark-provided seed inputs used to initialise fuzzing. We keep the tasks, prompts, sampled programs, and benchmark-provided initial inputs fixed, and rerun only the fuzzing step with three different random seeds. This isolates the effect of randomness in the mutation process itself. We re-run the LiveCodeBench pipeline on GPT-4o-mini (T = 0.6, K = N = 10) under three independent fuzz-input-generator seeds. As shown in Table 7, cross-seed standard deviations are at most 0.0024 on AUROC and 0.0050 on Spearman for both SDE and DSDE, indicating that the reported numbers do not depend on the choice of seed. Table 7: Robustness to fuzz-input-generator randomness on LiveCodeBench (GPT-4o-mini, T = 0.6, K = N = 10). The random seed controlling fuzz-input mutations is varied. Results report mean ± standard deviation over three independent fuzz-input-generator seeds.
5
Metric
AUROC
Pearson
Spearman
SDE DSDE
0.8238 ± 0.0011 0.8413 ± 0.0005
−0.5880 ± 0.0035 −0.6249 ± 0.0030
−0.5831 ± 0.0022 −0.6240 ± 0.0023
Limitations
Our framework inherits several limitations from its underlying assumptions. (i) Equivalence proxy. Effectiveness depends on the proxy exposing meaningful behavioural differences; weak fuzz inputs may yield overly coarse equivalence classes. (ii) Fuzzing is not a correctness oracle. Two incorrect programs may agree on all sampled inputs but diverge elsewhere, and two valid programs may disagree when the specification admits multiple acceptable behaviours, as seen on BigCodeBench. (iii) Top-ranked reference. When the top-ranked sample is unrepresentative of the model’s behavioural distribution, anchoring becomes unstable, consistent with our DeepSeek-Coder-V2 result where SDE slightly outperforms DSDE. (iv) Heuristic graded distance. The abnormal-execution costs (a, b, c) are hyperparameters with no canonical value across failure modes. Weights may need adaptation for other runtimes, languages, or benchmarks.
6
Conclusions
We introduced a four-axis taxonomy of sample-based uncertainty estimators (Table 1) and identified an unoccupied region: pairwise uncertainty estimation with graded semantic dissimilarity. We 9
instantiated this region with two estimators: SDE, a symmetric Rao quadratic entropy over execution clusters, and DSDE, a top-anchored variant measuring disagreement between alternative samples and the served output. Both improve predictive performance over sample-based baselines at lower computational cost, and remain stable across benchmarks, languages, temperatures and fuzzing settings.
References [1] Yasin Abbasi-Yadkori, Ilja Kuzborskij, András György, and Csaba Szepesvari. To believe or not to believe your llm: Iterative prompting for estimating epistemic uncertainty. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. [2] Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. Program synthesis with large language models. CoRR, abs/2108.07732, 2021. URL https://arxiv. org/abs/2108.07732. [3] Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. Pearson correlation coefficient. In Noise Reduction in Speech Processing, volume 2 of Springer Topics in Signal Processing, pages 1–37. Springer, Berlin, Heidelberg, 2009. ISBN 978-3-642-00296-0. doi: 10.1007/ 978-3-642-00296-0_5. [4] Andrew P. Bradley. The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognition, 30(7):1145–1159, 1997. ISSN 0031-3203. doi: 10.1016/S0031-3203(96)00142-2. [5] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code, 2021. URL https://arxiv.org/abs/2107.03374. [6] Vítor Mateus de Brito and Kleinner Farias. Understanding the role of large language models in software engineering: Evidence from an industry survey, 2025. URL https://arxiv.org/ abs/2512.21347. [7] Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625–630, 2024. doi: 10.1038/s41586-024-07421-0. [8] Robert Friel and Amartya Sanyal. Chainpoll: A high efficacy method for llm hallucination detection. arXiv preprint arXiv:2310.18344, 2023. URL https://arxiv.org/abs/2310. 18344. [9] Cuiyun Gao, Guodong Fan, Chun Yong Chong, Shizhan Chen, Chao Liu, David Lo, Zibin Zheng, and Qing Liao. A systematic literature review of code hallucinations in llms: Characterization, mitigation methods, challenges, and future directions for reliable ai, 2025. URL https: //arxiv.org/abs/2511.00776. [10] Patrice Godefroid, Michael Y Levin, David A Molnar, et al. Automated whitebox fuzz testing. In Ndss, volume 8, pages 151–166, 2008. [11] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Honghui Ding, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jingchang Chen, Jingyang Yuan, Jinhao Tu, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaichao You, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingxu Zhou, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi 10
Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-r1 incentivizes reasoning in large language models through reinforcement learning. Nature, 645(8081):633–638, 2025. [12] James A. Hanley and Barbara J. McNeil. The meaning and use of the area under a receiver operating characteristic (roc) curve. Radiology, 143(1):29–36, 1982. [13] Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with APPS, 2021. URL https://arxiv.org/abs/2105.09938. [14] Yuheng Huang, Jiayang Song, Zhijie Wang, Shengming Zhao, Huaming Chen, Felix Juefei-Xu, and Lei Ma. Look before you leap: An exploratory study of uncertainty analysis for large language models. IEEE Transactions on Software Engineering, 51(2):413–429, 2025. doi: 10.1109/TSE.2024.3519464. [15] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL https://openreview.net/forum?id=chfJJYC3iL. [16] Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openreview.net/forum?id=VD-AYtP0dve. [17] Jingxuan Li, Yuxin Zhu, Yiming Li, Guoping Li, and Zhi Jin. Showing llm-generated code selectively based on confidence of llms. arXiv preprint arXiv:2410.03234, 2024. URL https: //arxiv.org/abs/2410.03234. [18] Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Generating with confidence: Uncertainty quantification for black-box large language models. arXiv preprint arXiv:2305.19187, 2023. [19] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation, 2023. URL https://arxiv.org/abs/2305.01210. [20] Potsawee Manakul, Ankit Liusie, and Mark J. F. Gales. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint arXiv:2303.08896, 2023. URL https://arxiv.org/abs/2303.08896. [21] Nhan Nguyen and Sarah Nadi. An empirical evaluation of github copilot’s code suggestions. In Proceedings of the 19th International Conference on Mining Software Repositories (MSR), pages 1–5, 2022. doi: 10.1145/3524842.3528470. [22] Rodrigo Pato Nogueira, Marco Vieira, and João R. Campos. Beyond functional correctness: An empirical evaluation of large language models for text-to-code generation. In Proceedings of the IEEE International Symposium on Software Reliability Engineering (ISSRE), pages 264–275, 2025. doi: 10.1109/ISSRE66568.2025.00036. 11
[23] Arindam Sharma and Cristina David. Assessing correctness in LLM-based code generation via uncertainty estimation, 2025. URL https://arxiv.org/abs/2502.11620. [24] Thomas Valentin, Ardi Madadi, Gaetano Sapia, and Marcel Böhme. Incoherence as oracle-less measure of error in LLM-based code generation. In Proceedings of the 40th AAAI Conference on Artificial Intelligence (AAAI), 2026. Accepted for publication. [25] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openreview.net/ forum?id=1PL1NIMMrw. [26] Clark Wissler. The spearman correlation formula. Science, 22(558):309–311, 1905. doi: 10.1126/science.22.558.309. [27] Sangyeop Yeo, Yu-Seung Ma, Sang Cheol Kim, Hyungkook Jun, and Taeho Kim. Framework for evaluating code generation ability of large language models. ETRI Journal, 46(1):106–117, 2024. doi: 10.4218/etrij.2023-0357. [28] Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Lei Shen, Zihan Wang, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x. In Ambuj K. Singh, Yizhou Sun, Leman Akoglu, Dimitrios Gunopulos, Xifeng Yan, Ravi Kumar, Fatma Ozcan, and Jieping Ye, editors, Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2023, Long Beach, CA, USA, August 6-10, 2023, pages 5673–5684. ACM, 2023. doi: 10.1145/3580305.3599790. URL https://doi.org/10.1145/3580305. 3599790. [29] Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, James Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, and et al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL https://openreview.net/forum?id=YrycTjllL0.
12
A
Detailed Input-Quality Diagnostics
The reliability of SDE and DSDE ultimately depends on whether the generated test inputs I produce useful behavioural signal on the sampled programs. This appendix reports four diagnostics that characterise this signal, computed per task over I = {x1 , . . . , x10 } and the candidate set C, and averaged across tasks. We organise the diagnostics into two groups: (i) input-side validity metrics that measure how well I is constructed; (ii) coverage metrics that measure how thoroughly I exercises the candidate code. Input validity metrics ValidExecRate is the fraction of inputs on which at least one candidate program returns a non-crash output. An input on which every candidate crashes carries no behavioural signal and cannot contribute to clustering: ValidExecRate =
|{ xj ∈ I : ∃ c ∈ C, c(xj ) is non-crash }| . |I|
(5)
UniqueInputRate is the fraction of inputs that remain distinct after canonical normalisation. A value below one indicates that the fuzzer has produced redundant or equivalent inputs within the same task: UniqueInputRate =
unique({normalize(xj ) : xj ∈ I}) . |I|
(6)
CrashPollutionRate is the fraction of all candidate-by-input execution cells that crash. A high rate indicates that many cells contribute no meaningful behavioural signal and therefore add noise to clustering: |{ (c, xj ) ∈ C × I : c(xj ) crashes }| . (7) CrashPollutionRate = |C| · |I| Code-coverage metrics LineCoverage and BranchCoverage measure the fraction of executable lines and branches in the candidate programs that are exercised by I. For each candidate c ∈ C, let L(c) and Lcov (c) denote the total and covered executable lines when c is executed on I, and let B(c) and Bcov (c) denote the analogous quantities for branches. The micro-averaged coverages are P P c∈C Bcov (c) c∈C Lcov (c) LineCoverage = P , BranchCoverage = P . (8) c∈C L(c) c∈C B(c) Together they bound how much of each candidate’s behaviour is observable through the fuzz suite. Table 8: Input-quality diagnostics on LiveCodeBench at T = 0.6. Per task we use the first N = 10 available inputs; tasks with no available inputs are excluded. Each metric is averaged over the remaining tasks. LineCov and BranchCov are micro-averaged. Model GPT-3.5-Turbo GPT-4o-mini Gemini-2.5-Flash-Lite Claude-opus-4.5
ValidExec Unique Crash LineCov BranchCov 0.901 0.898 0.851 0.863
0.948 0.948 0.951 0.950
0.209 0.213 0.269 0.201
0.937 0.937 0.919 0.917
0.888 0.889 0.874 0.879
Results. Table 8 reports the diagnostics on LiveCodeBench at T = 0.6. UniqueInputRate is approximately 0.95 across all four models, and ValidExecRate stays above 0.85, indicating that the fuzzer rarely produces redundant inputs and that most inputs successfully exercise at least one candidate program. LineCoverage exceeds 0.91 and BranchCoverage exceeds 0.87 on every model, indicating that the fuzz suite reaches most code paths in each candidate. The input pool therefore remains diverse, mostly 13
executable, and largely covering across all four models. CrashPollutionRate sits around 0.20–0.27, indicating that roughly one in five execution cells contributes no behavioural signal; reducing this rate (e.g. via type-aware input constraints or LLM-guided fuzzers) is a direct lever for strengthening the upstream signal that SDE and DSDE consume.
B
End-to-End Evaluation Pipeline
Figure 3 summarises the end-to-end evaluation pipeline used in this paper. Candidate programs are first sampled from the code generation model; type-aware mutations then produce a shared set of inputs; each candidate is executed on this input set to obtain an execution signature; programs with identical signatures are clustered; and finally the distance-aware uncertainty metrics SDE and DSDE are computed from the resulting cluster distribution.
Figure 3: End-to-end evaluation pipeline for execution-based uncertainty estimation in code generation.
C
Extended Experimental Results
This appendix presents additional experimental analyses that complement the main evaluation in Section 4. Unless stated otherwise, all settings follow the protocol used in the main paper: K = 10 sampled programs and N = 10 test inputs per task, with execution timeout 0.2 s and sampling temperature T = 0.6. C.1
Extended Per-Difficulty Tables
Table 9 and Table 10 correspond directly to the AUROC and correlation tables in the main paper, evaluated on four closed-source code-generation models — GPT-3.5-Turbo-0125, GPT-4o-mini, Gemini-2.5-Flash-Lite, and Claude-opus-4.5 — with results for an additional open-source model (DeepSeek-Coder-V2) reported in Section 4.3. We compare our two distance-aware uncertainty metrics, SDE and DSDE, against three sample-based uncertainty baselines: DiffTrust [24], an incoherence-based behavioural-disagreement metric for code generation; HonestCoder [17], which combines semantic embeddings and dataflow features with syntactic n-gram features over sampled programs; and Semantic Entropy [7], which clusters programs based on binary semantic-equivalence judgments from an LLM judge and computes Shannon entropy over the resulting clusters. Discussion of the Claude–Easy outlier. On the LCB Easy split for Claude-opus-4.5 the AUROC drops markedly relative to the other splits. This appears to be driven by extreme class imbalance: Claude achieves a pass@1 success rate of 95.34% on this subset, leaving very few failure instances. AUROC is equivalent to the Wilcoxon rank-sum statistic [12], whose standard error grows rapidly as the minority class shrinks, so small changes in the ranking of a handful of failures produce disproportionate fluctuations in the reported value. 14
Table 9: AUROC for pass@1 failure prediction across task difficulty levels and models (T = 0.6). SDE
DSDE
DiffTrust
HonestCoder Semantic Entropy
0.827 0.786 0.718 0.806
0.844 0.801 0.753 0.831
0.669 0.577 0.610 0.568
0.646 0.645 0.576 0.570
0.605 0.592 0.548 0.550
0.826 0.785 0.712 0.801
0.844 0.796 0.752 0.829
0.534 0.565 0.504 0.523
0.646 0.645 0.576 0.570
0.607 0.594 0.548 0.553
0.794 0.735 0.742 0.705
0.808 0.730 0.762 0.737
0.688 0.671 0.686 0.523
0.668 0.566 0.635 0.590
0.639 0.470 0.597 0.645
0.818 0.483 0.723 0.815
0.825 0.482 0.733 0.839
0.606 0.473 0.556 0.507
0.739 0.423 0.605 0.735
0.687 0.452 0.572 0.638
GPT-3.5-Turbo-0125 LiveCodeBench All LiveCodeBench Easy LiveCodeBench Medium LiveCodeBench Hard GPT-4o-mini LiveCodeBench All LiveCodeBench Easy LiveCodeBench Medium LiveCodeBench Hard Gemini-2.5-Flash-Lite LiveCodeBench All LiveCodeBench Easy LiveCodeBench Medium LiveCodeBench Hard Claude-opus-4.5 LiveCodeBench All LiveCodeBench Easy LiveCodeBench Medium LiveCodeBench Hard
C.2
Sensitivity to Distance Weights
The crash-aware distance weights (a, b, c) are fixed to (1, 0.8, 0.6) throughout the main paper. To quantify the gap between this fixed scheme and a per-setting tuned alternative, we additionally consider learning (a, b, c) via an 80:20 train–test split: the weights are fitted on the training set and the resulting AUROC is measured on the held-out test set. The default and learned schemes differ by less than 0.011 in AUROC across all settings. On LiveCodeBench, only a small subset of inputs (consistently below 3 out of 10) result in abnormal execution and therefore depend on (a, b, c), which explains the small gap. In domains where abnormal executions are more frequent, learning the weights is expected to become more beneficial; for the regimes considered here, the fixed weighting scheme is sufficient. C.3
Benchmark Details
LiveCodeBench (LCB) is our primary benchmark, containing 1,055 programming tasks stratified into Easy, Medium, and Hard difficulty levels with a relatively balanced distribution. MBPP contains 257 entry-level Python function-completion problems. HumanEval-X comprises 164 hand-written problems parallel-translated into Python, Java, and C++, and is used to test cross-language transfer. BigCodeBench consists of 1,140 repository-scale tasks featuring cross-file dependencies, multiple interacting components, and complex control flow. Together these benchmarks span a wide range of task difficulty (from entry-level to repository-scale), programming languages (Python, Java, C++), and problem styles (function completion, competitive programming, and realistic library usage). C.4
Abstention Policy for Practical Deployment
To support practical use of uncertainty estimates, we introduce a simple abstention policy that determines whether to accept or reject the model’s top-ranked generated solution. The policy requires no additional models or supervision; instead, it operationalises our distance-aware uncertainty metrics as an actionable decision rule. 15
Table 10: Correlation with partial_pass@1 across task difficulty levels and models (T = 0.6). Left block: Pearson r. Right block: Spearman ρ. Pearson r
Spearman ρ
SDE
DSDE DiffTrust
HC
Semantic Entropy
SDE
DSDE DiffTrust
HC
Semantic Entropy
-0.587 -0.588 -0.412 -0.474
-0.622 -0.645 -0.467 -0.516
-0.259 -0.184 -0.153 -0.063
-0.344 -0.259 -0.185 -0.205
-0.240 -0.169 -0.112 -0.144
-0.581 -0.549 -0.426 -0.429
-0.620 -0.581 -0.474 -0.492
-0.267 -0.145 -0.144 -0.044
-0.328 -0.256 -0.179 -0.162
-0.237 -0.176 -0.105 -0.121
-0.589 -0.590 -0.416 -0.470
-0.624 -0.644 -0.474 -0.515
-0.024 -0.180 0.013 -0.018
-0.344 -0.259 -0.185 -0.205
-0.240 -0.170 -0.110 -0.142
-0.584 -0.547 -0.433 -0.426
-0.624 -0.571 -0.488 -0.485
-0.064 -0.143 -0.033 -0.059
-0.328 -0.255 -0.179 -0.162
-0.238 -0.176 -0.106 -0.123
-0.494 -0.414 -0.446 -0.307
-0.532 -0.449 -0.499 -0.370
-0.296 -0.307 -0.346 0.025
-0.301 -0.106 -0.249 -0.221
-0.237 -0.005 -0.179 -0.237
-0.491 -0.314 -0.414 -0.308
-0.512 -0.308 -0.443 -0.365
-0.298 -0.215 -0.309 0.031
-0.329 -0.134 -0.275 -0.184
-0.242 -0.033 -0.178 -0.235
LiveCodeBench All -0.552 -0.611 LiveCodeBench Easy 0.005 0.026 LiveCodeBench Medium -0.406 -0.492 LiveCodeBench Hard -0.548 -0.635
-0.139 -0.056 -0.082 0.008
-0.424 0.071 -0.177 -0.446
-0.275 0.057 -0.099 -0.252
-0.450 -0.461 0.024 0.025 -0.253 -0.265 -0.531 -0.580
-0.139 0.024 -0.059 -0.003
-0.350 0.052 -0.143 -0.414
-0.287 0.048 -0.107 -0.273
GPT-3.5-Turbo-0125 LiveCodeBench All LiveCodeBench Easy LiveCodeBench Medium LiveCodeBench Hard GPT-4o-mini LiveCodeBench All LiveCodeBench Easy LiveCodeBench Medium LiveCodeBench Hard Gemini-2.5-Flash-Lite LiveCodeBench All LiveCodeBench Easy LiveCodeBench Medium LiveCodeBench Hard Claude-opus-4.5
Table 11: Fixed default weights (1.0, 0.8, 0.6) vs. per-setting learned weights. (a, b, c)
Default
Learned
∆
Abnormal (/10)
(0.90, 0.90, 0.25) (0.90, 0.90, 0.90) (0.97, 0.88, 0.88) (1.00, 0.80, 0.80)
0.8248 0.8075 0.8442 0.8442
0.8206 0.7974 0.8395 0.8524
−0.0042 −0.0101 −0.0047 +0.0082
2.01 2.69 2.09 2.13
0.7789 0.8442 0.8530
0.7743 0.8524 0.8446
−0.0045 +0.0082 −0.0084
2.69 2.13 1.82
Setting Across models (fixed T = 0.6) Claude Opus 4.5 Gemini-2.5-Flash-Lite GPT-3.5-Turbo GPT-4o-mini
Across temperatures (fixed GPT-4o-mini) GPT-4o-mini (T = 0.2) GPT-4o-mini (T = 0.6) GPT-4o-mini (T = 0.8)
(0.51, 0.41, 0.31) (1.00, 0.80, 0.80) (0.60, 0.60, 0.60)
Formally, given a programming task π and its associated uncertainty score U (π), the abstention decision is defined as 1, U (π) ≤ τ (accept) δ(π) = 0, U (π) > τ (abstain), where τ is a decision threshold selected via cross-validation on training data and evaluated on held-out validation folds. Table 13 reports abstention performance for GPT-4o-mini under different operating constraints. In safety-oriented settings, we impose an upper bound on the false positive rate (FPR), defined as the fraction of incorrect solutions that are accepted. For each constraint, τ is chosen to maximise accuracy under strict correctness (Pass@1 = 1). Under a highly conservative constraint (FPR ≤ 5%), both SDE and DSDE achieve strong and comparable performance. Under a more balanced setting (FPR ≤ 20%), DSDE yields a consistent accuracy improvement over SDE. 16
Table 12: Statistics of the four benchmarks used in this paper. Benchmark
#Tasks
Languages
Task style
Difficulty
LiveCodeBench [15] MBPP [2] HumanEval-X [28] BigCodeBench [29]
1,055 257 164 1,140
Python Python Python, Java, C++ Python
Competitive programming Function completion Function completion Repository-scale, library usage
Easy / Medium / Hard Entry-level Mixed Hard
Table 13: Abstention policy performance under strict correctness (Pass@1 = 1.0) on LiveCodeBench using GPT-4o-mini (T = 0.6). Results are shown under two representative operating points: Strict Safety and Balanced. Accuracy is maximised subject to the corresponding FPR constraint. Values are reported as mean ± standard deviation over 5-fold cross-validation.
C.5
Metric
Constraint
Accuracy
FPR
SDE DSDE
FPR ≤ 5% FPR ≤ 5%
0.745 ± 0.095 0.745 ± 0.095
0.064 ± 0.028 0.064 ± 0.028
SDE DSDE
FPR ≤ 20% FPR ≤ 20%
0.734 ± 0.088 0.750 ± 0.095
0.069 ± 0.028 0.068 ± 0.028
Execution Timeout
Performance is robust to the choice of per-input execution timeout (Table 14). Across all settings, the proportion of LiveCodeBench problems that triggered at least one timeout error is below 2%, indicating that the timeout has only a minor effect on abnormal-termination rates. Table 14: Effect of the per-input execution timeout on SDE and DSDE (GPT-4o-mini, T = 0.6). Pearson r
AUROC
Spearman ρ
Timeout (s)
SE
SDE
DSDE
SE
SDE
DSDE
SE
SDE
DSDE
0.1 0.2 0.3
0.802 0.824 0.798
0.804 0.826 0.808
0.819 0.844 0.821
−0.505 −0.543 −0.499
−0.540 −0.589 −0.550
−0.575 −0.624 −0.583
−0.493 −0.524 −0.486
−0.534 −0.584 −0.545
−0.573 −0.624 −0.580
17
D
Notation
Table 15 lists every symbol used in the paper. Where the pipeline (§3) and metrics (§2) sections use different surface forms for the same object, both forms are shown joined by ‘≡’. Table 15: Notation summary. Defaults from §4.1 are in parentheses. Symbol
Description
(i) Task and model d Coder(·) p(π | d)
Natural-language task description Code-generation model Conditional distribution over programs given d
(ii) Candidate programs K π, πk , π1 π≡Π
Number of candidates per task (K=10) Generic / k-th sampled / top-ranked candidate Candidate set {π1 , . . . , πK }
(iii) Test inputs and execution N iℓ ≡ x ℓ i≡I π(iℓ ) ≡ oℓ (π) ok ≡ σI (πk )
Number of test inputs per task (N =10) The ℓ-th test input, ℓ ∈ {1, . . . , N } Input set (i1 , . . . , iN ), shared across candidates Outcome of π on iℓ : a normal value or labelled error Execution signature o1 (πk ), . . . , oN (πk )
(iv) Behaviour clusters M Ci , |Ci | C(π) pi c∗
Number of clusters The i-th cluster and its size Cluster containing program π P Empirical cluster probability |Ci |/K, i pi =1 Index of cluster containing π1 , i.e. Cc∗ = C(π1 )
(v) Distances and weights δ(·, ·) dij a b c
Per-input outcome dissimilarity (Eq. (2)), δ ∈ [0, 1] Cluster-level distance (Eq. (1)); symmetric, dii =0 Cost when exactly one outcome is abnormal (a=1) Cost when both outcomes abnormal, different errors (b=0.8) Cost when both outcomes abnormal, same error (c=0.6)
(vi) Uncertainty metrics SDE DSDE U (π)
Semantic Distance Entropy (Eq. (3)) Dominant SDE, anchored at Cc∗ (Eq. (4)) Generic uncertainty score (e.g., SDE or DSDE) used in abstention
(vii) Experimental settings and targets T τ FPR pass@1 partial_pass@1 AUROC r, ρ
Sampling temperature (T =0.6) Decision threshold of the abstention policy False positive rate (incorrect solutions accepted) Binary: π1 passes all reference tests Fraction of reference tests passed by π1 , in [0, 1] Area under ROC for predicting pass@1 failure Pearson and Spearman correlations with partial_pass@1
18
E
Concrete Case Study Examples
This appendix provides full task descriptions and representative cluster implementations for the two case studies. Each task is sampled K = 10 times; programs sharing identical execution outcomes on the shared test inputs are grouped into a single execution-behaviour cluster. The cluster membership for all four tasks is summarised in Table 16. To complement the results in RQ1, we present two representative case studies that explicitly isolate the effect of semantic distance between execution clusters. The first case study focuses on binary correctness (pass@1), while the second examines partial correctness (partial_pass@1). In both cases, the tasks induce the same number of execution clusters (i.e., the same count of distinct behaviours); differences in correctness therefore arise solely from variation in the semantic distance between clusters rather than from the number of observed behaviours. Table 16: Execution-behaviour cluster membership and cluster probabilities for the case-study tasks. Task ID
Cluster
Cluster prob.
Member program indices
0.800 (8/10) 0.200 (2/10) 0.200 (2/10) 0.800 (8/10)
[0, 1, 2, 3, 4, 5, 6, 7] [8, 9] [0, 3] [1, 2, 4, 5, 6, 7, 8, 9]
Case A: Binary correctness distinction 3367 3367 abc332_b abc332_b
C0 C1 C0 C1
Case B: Partial correctness under identical pass@1 abc326_b abc326_b 3163 3163
E.1
C0 C1 C0 C1
0.400 (4/10) 0.600 (6/10) 0.400 (4/10) 0.600 (6/10)
[0, 4, 5, 7] [1, 2, 3, 6, 8, 9] [0, 1, 3, 8] [2, 4, 5, 6, 7, 9]
Case A: Binary Correctness Distinction
Case A contrasts two LiveCodeBench tasks whose execution-behaviour cluster structures are identical (in both tasks the 10 sampled programs split into two clusters of size 8 and 2), yet whose binary correctness outcomes differ. The contrast isolates the contribution of inter-cluster semantic distance: dispersion-based statistics cannot distinguish the two cases, but the distance-aware metrics can. Insight: distance-aware uncertainty LiveCodeBench · Case Study pass@1 = 1 ✓
LCB Task 3367
LCB Task abc332_b pass@1 = 0 ✗
Find Sum of Encrypted Integers
Glass and Mug
10 programs → 2 clusters (8 / 2)
10 programs → 2 clusters (8 / 2) d
d
C0 : 8 progs (correct)
distance
C1 : 2 fail neg.
d = 0.10 0.0160
SDE DSDE
C1 : 8 wrong order
0.0200 ✓ low uncertainty / accept
distance
C0 : 2 wrong cond.
d = 1.00 0.1600
SDE DSDE
0.8000 ✗ high uncertainty / abstain
Same cluster structure, different inter-cluster distance: SDE and DSDE both capture this. On the failing case, DSDE amplifies SDE's signal 5× (0.80 vs 0.16) for sharper uncertainty.
Figure 4: Case A: two LiveCodeBench tasks with identical execution-behaviour cluster structures but different inter-cluster semantic distances and binary correctness outcomes. Case A: Semantic Distance Explains Binary Correctness (pass@1). Table 17 compares two tasks with identical cluster structure (M = 2, one cluster contains 2 programs, and the other 8 programs) but different binary outcomes. One task consistently produces correct executions (pass@1 = 1), while the other fails (pass@1 = 0). Importantly, both tasks exhibit the same dispersion over execution clusters, ruling out dispersion-based explanations. 19
Table 17: Case A: Metric-level comparison illustrating the binary correctness distinction. Task 1 (3367, pass@1 = 1) and Task 2 (abc332_b, pass@1 = 0) induce an identical M = 2 cluster structure with sizes 8 and 2, but exhibit very different inter-cluster distances. Metric
Task 1 (Low-Distance)
Task 2 (High-Distance)
Diff.
Pass@1 partial_pass@1
1 1.0000
0 0.0000
– 1.0000
Clusters Distance
2 (8/2) 0.10
2 (8/2) 1.00
0 0.90
SDE DSDE
0.0160 0.0200
0.1600 0.8000
0.1440 0.7800
Although the two tasks induce identical cluster structures, they differ substantially in the semantic distance between clusters. For the correct task (task 1), the distance between the two clusters is small (e.g., d01 = 0.10), indicating highly similar execution outcomes. In contrast, the incorrect task (task 2) exhibits a much larger inter-cluster distance (e.g., d01 = 1.00), reflecting severe semantic divergence. This difference in cluster distance directly explains the observed binary correctness gap and is captured by distance-aware uncertainty metrics such as SDE and DSDE. E.1.1
Problem 3367: Find the Sum of Encrypted Integers
Problem statement. Given an integer array nums, define encrypt(x) as replacing every digit in x with the largest digit in x. Return the sum of encrypted elements. For example, encrypt(523) = 555 and encrypt(213) = 333. Cluster structure. Two clusters with probabilities p0 = 0.80 (8 samples) and p1 = 0.20 (2 samples). Cluster C0 passes all test cases; cluster C1 fails on negative-number inputs with a ValueError. 1 2 3 4 5 6 7 8
1 2 3 4 5 6 7 8
Cluster C0 — Correct implementation (8 samples)
Key characteristic
class Solution: def sumOfEncryptedInt(self, nums: List[int]) -> int: total_sum = 0 for num in nums: max_digit = max(str(num)) encrypted_num = int(max_digit * len(str(num))) total_sum += encrypted_num return total_sum
Uses max(str(num)), which returns the maximum character directly and handles positive integers correctly.
Cluster C1 — Fails on negative numbers (2 samples)
Key characteristic
class Solution: def sumOfEncryptedInt(self, nums: List[int]) -> int: total_sum = 0 for num in nums: largest_digit = max([int(digit) for digit in str( num)]) encrypted_num = int(str(largest_digit) * len(str( num))) total_sum += encrypted_num return total_sum
Attempts to convert each character to an integer, raising ValueError: invalid literal for int() with base 10: ’-’ on negative inputs.
E.1.2
Problem abc332_b: Glass and Mug
Problem statement. Given a glass with capacity G ml and a mug with capacity M ml, both initially empty, repeat K operations: (1) if the glass is empty, fill the mug completely; (2) else if the mug is empty, discard all water from the glass; (3) otherwise, transfer water from the mug to the glass (the glass fills completely if possible, with any remainder staying in the mug). Output the final water amounts in the glass and the mug. Cluster structure. Two clusters with probabilities p0 = 0.20 (2 samples) and p1 = 0.80 (8 samples). Both clusters produce incorrect solutions, but with different error patterns. 20
1 2 3 4 5 6 7 8 9 10 11 12
1 2 3 4 5 6 7 8 9 10 11 12
Cluster C0 — Wrong condition check (2 samples)
Key characteristic
K, G, M = map(int, input().split()) glass, mug = 0, 0 for _ in range(K): if glass < G: # Wrong: should check glass == 0 glass = G elif mug == 0: mug = M else: transfer = min(mug, G - glass) glass += transfer mug -= transfer print(glass, mug)
Uses glass < G instead of glass == 0, misinterpreting the problem requirements.
Cluster C1 — Incorrect operation order (8 samples)
Key characteristic
K, G, M = map(int, input().split()) glass, mug = 0, 0 for _ in range(K): if glass == 0: mug = M elif mug == 0: glass = 0 # Wrong: should fill glass, not empty it else: transferred = min(M - mug, glass) mug += transferred glass -= transferred print(glass, mug)
The glass is emptied when the mug is empty (line 6), so the glass always ends at 0.
E.2
Case B: Partial Correctness Distinction
Case B isolates the contribution of inter-cluster distance under identical pass@1 outcomes: both tasks have pass@1 = 0 and admit the same 6/4 cluster split, yet differ substantially in partial_pass@1. Distance-aware metrics recover this distinction; dispersion-only metrics cannot. Case B: Semantic Distance Explains Partial Correctness. Table 18 presents a complementary scenario, where both tasks have pass@1 = 0 and induce identical execution cluster structure (M = 2, one cluster contains 6 programs, and the other 4 programs), yet differ substantially in partial_pass@1. Table 18: Case B: Metric-level comparison illustrating partial correctness under identical binary outcomes. Task 1 (abc326_b) and Task 2 (3163) both have pass@1 = 0 and induce an identical M = 2 cluster structure with sizes 6 and 4, but differ in their inter-cluster distance and therefore in partial_pass@1. Metric
Task 1 (Low-Distance)
Task 2 (High-Distance)
Diff.
pass@1 partial_pass@1
0 0.6667
0 0.0000
– 0.6667
Clusters Distance
2 (6/4) 0.30
2 (6/4) 1.00
0 0.70
SDE DSDE
0.0720 0.1800
0.2400 0.6000
0.1680 0.4200
In this case, dispersion-based signals provide no discrimination, as the two tasks share identical cluster structures and probabilities. However, the semantic distance between clusters differs markedly. The task with higher partial correctness exhibits smaller inter-cluster distances (e.g., d01 = 0.30), whereas the task with lower partial correctness shows much larger distances (e.g., d01 = 1.00). These differences align closely with the partial correctness outcomes and are faithfully reflected by distance-aware metrics. E.2.1
Problem abc326_b: 326-like Numbers
Problem statement. A positive integer is a 326-like number if it is a three-digit number satisfying (hundreds digit) × (tens digit) = (ones digit). For example, 326, 400, and 144 are 326-like. Given a positive integer N , return the smallest integer ≥ N that is a 326-like number. 21
Cluster structure. Two clusters with probabilities p0 = 0.60 (6 samples, partial_pass@1 = 1.0) and p1 = 0.40 (4 samples, partial_pass@1 = 0.0). The two clusters differ by an off-by-one ordering of the increment and the predicate check. 1 2 3 4 5 6 7 8 9
1 2 3 4 5 6 7 8 9
Cluster C0 — Correct: check before increment (6 samples)
Key characteristic
N = int(input()) while True: hundreds = N // 100 tens = (N % 100) // 10 ones = N % 10 if hundreds * tens == ones: print(N) break N += 1
The predicate is checked before incrementing N , so the case in which N itself is already a 326-like number is handled correctly. # Check condition first
# Then increment
Cluster C1 — Off-by-one error (4 samples)
Key characteristic
N = int(input()) while True: N += 1 checking hundreds = N // 100 tens = (N % 100) // 10 ones = N % 10 if hundreds * tens == ones: print(N) break
N is incremented before the predicate is checked, producing an off-by-one error whenever N itself is already a 326-like number.
E.2.2
# Wrong: increment before
# Then check condition
Problem 3163: Subarrays Distinct Element Sum of Squares II
Problem statement. Given a 0-indexed integer array nums, the distinct count of a subarray is the number of distinct elements it contains. Return the sum of the squares of the distinct counts of all subarrays. For example, with nums = [1,2,1], the answer is 12 + 12 + 12 + 22 + 22 + 22 = 15. Cluster structure. Two clusters with probabilities p0 = 0.40 (4 samples, partial_pass@1 = 0.0) and p1 = 0.60 (6 samples, partial_pass@1 = 1.0). The two clusters differ in what is being squared. 1 2 3 4 5 6 7 8 9 10 11 12
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Cluster C0 — Wrong: sums squared frequencies (4 samples)
Key characteristic
from collections import Counter class Solution: def sumCounts(self, nums: List[int]) -> int: total = 0 n = len(nums) for i in range(n): count = Counter() for j in range(i, n): count[nums[j]] += 1 # Wrong: sums (frequency^2), not (distinct count)^2 total += sum(val ** 2 for val in count.values ()) return total
P Computes k fk2 , where fk is the frequency of element k, instead of (number of distinct elements)2 —a fundamental misinterpretation of the specification.
Cluster C1 — Correct: squares the distinct count (6 samples)
Key characteristic
class Solution: def sumCounts(self, nums: List[int]) -> int: ans = 0 n = len(nums) for i in range(n): freq = {} distinct_count = 0 for j in range(i, n): if nums[j] not in freq: distinct_count += 1 freq[nums[j]] = 1 else: freq[nums[j]] += 1 ans += distinct_count ** 2 # Correct: ( distinct count)^2 return ans
Tracks the number of distinct elements and squares this count, matching the specification.
22
Across both cases, correctness differences cannot be explained by cluster counts or dispersion alone. Instead, they are driven by the magnitude of semantic distance between execution clusters. These examples provide concrete, intuitive evidence that the effectiveness observed in Section 4.2 arises from incorporating semantic distance, supporting the use of distance-aware uncertainty estimation for LLM-generated code.
23