Code Is More Than Text: Uncertainty Estimation for Code Generation Yuling Shi1, * Caiqi Zhang2, * Yuexian Li1 Haopeng Wang1 Yeheng Chen1 Nigel Collier2 Xiaodong Gu1 1
Shanghai Jiao Tong University
Abstract
arXiv:2606.09577v1 [cs.CL] 8 Jun 2026
University of Cambridge
deployment, gating selective prediction in autocomplete, prioritizing human review in code agents, and informing retry and escalation policies (Zhang et al., 2024). However, the methods that currently dominate code UE are direct ports of NL UE, including length-normalized likelihood, mean predictive entropy, semantic-consistency clustering, and verbalized confidence prompts (Malinin and Gales, 2021; Kuhn et al., 2023; Sharma and David, 2025; Zhang et al., 2024), which treat code as just another token sequence. We argue that this abstraction fails to exploit the characteristics of code. Code differs from NL in three under-exploited ways. First, code suffers from token fragility: a single wrong operator can break the entire program, so uncertainty in code is sparse rather than uniform. Second, a program has an intent–code gap that separates algorithmic intent from concrete implementation; code can be abstracted into pseudo-code, which strips away surface variation and provides a cleaner basis for consistency-based estimation than raw code. Third, code is executable: the pass/fail pattern of a candidate program against test cases is a direct behavioral signal of functional correctness: a signal that is unique in NL generation. How to design a UE method that respects these three properties remains an open question. Motivated by these properties, we propose treating code uncertainty as a three-axis quantity in which each axis captures exactly one property (Figure 1, Contribution #1): a lexical axis (Top-K token entropy) for token fragility, an algorithmic axis (pseudo-code consistency) that elicits N natural-language solution plans and measures their inter-sample agreement at the algorithmic level, and a functional axis (behavioral consistency) that scores how consistently a candidate’s runtime behavior agrees with the self-generated test cases, providing a calibration signal rather than a selection mechanism, in contrast to CodeT-style rerank-
Large language models (LLMs) are increasingly deployed as code generators, where silently wrong programs pose real safety and reliability risks. Reliable uncertainty estimation (UE) is essential for selective prediction, human-in-the-loop review, and downstream agentic decisions. Yet most existing code UE methods are inherited from natural language (NL) generation and ignore properties that make code distinct. We argue that code differs from NL in three ways: a single wrong token can break an entire program (token fragility); algorithmic intent and concrete implementation can disagree independently (intent–code gap); and programs can be executed (executability). We instantiate these properties as three orthogonal uncertainty axes: lexical (Top-K token entropy), algorithmic (pseudo-code consistency), and functional (behavioral consistency). Across five code LLMs, our threeaxis ensemble improves average AUROC from 0.696 for the strongest NL-derived baseline to 0.776 (+8.1 points). Notably, on Qwen3-14B, our single-pass Top-K token entropy matches the strongest multi-pass baseline while being over 3× cheaper; across models, it remains a competitive low-cost signal. These results suggest that code UE deserves code-specific design rather than direct NL ports.
1
2
Introduction
Large language models (LLMs) have moved from single-line code completion to acting as the execution unit of IDE assistants, autonomous coding agents, and multi-step software-engineering pipelines (Chen et al., 2021; Austin et al., 2021; Shi et al., 2025c). A silently wrong output is far more dangerous than a refusal or a flagged guess: errors slip past review and compound across downstream steps. Reliable uncertainty estimation (UE) has accordingly become increasingly important to safe * Equal contribution.
1
1
Lexical Uncertainty
2
Algorithmic Uncertainty
3
Functional Uncertainty
“Which tokens are most uncertain?”
“Which is the strategy behind this code?”
“What is the code’s accuracy on test cases?”
Token Level
Strategy Level
Execution Level
for i, x in enumerate(nums):
for x in nums: if target - x in seen:
need = target - x if need in seen:
return True seen.add(x)
return seen[need], i seen[x] = i
Step 1 Compute complement: target - x
High-entropy tokens: target
if
Top-K high-entropy tokens
return
Uncertainty signal
Few High-Uncertainty Tokens
Step 2 Check whether complement appeared before Step 3 Update memory set: seen.add(x) Step match
ROUGE-L
Bidirectional avg
Two-Level Structural Uncertainty
Test Case (Input)
Expected
Predicted
nums = [2, 7, 11, 15], target = 9
True
True
Result Pass
nums = [3, 2, 4], target = 6
True
True
Pass
nums = [1, 2, 3], target = 7
False
False
Pass
nums = [1, 1], target = 2
True
False
Fail
nums = [], target = 1
False
True
Fail
Pass rate: 3 / 5 = 60% Failure rate (Uncertainty): 40%
Behavioral (Execution) Uncertainty
Figure 1: Three methods based on characteristics of coding uncertainty signals
ing (Chen et al., 2023). The three axes are designed to be complementary, and a simple ranknormalized weighted sum gives an ensemble whose role is to demonstrate that complementarity rather than to be the contribution itself. We evaluate on four benchmarks (APPS-Intro, APPS-Interview, HumanEval, MBPP) across five code-capable LLMs, comparing against single- and multi-pass NL UE baselines (Contribution #2). Our findings are as follows. First, each of the three axes independently matches or outperforms the strongest NL baseline on the property it targets, confirming that code-specific signals carry information that NL-derived methods miss (§4). Second, on Qwen3-14B, Top-K token entropy alone, despite being single-pass, matches the strongest multipass NL baseline at over 3× lower cost; across models, it remains a competitive low-cost signal, indicating that token-level entropy has been substantially undervalued for code. Combining the three signals further improves performance, with the ensemble reaching 0.800 average AUROC on Qwen3-14B and 0.776 average AUROC across all five models (§5). Third, splitting token entropy over code versus comment tokens reveals an asymmetry invisible to full-sequence aggregation: code-only AUROC is 0.716, while commentonly entropy is worse than random (0.375) (§5). A cost–performance analysis (§5, Contribution #3) recommends Top-K entropy under tight sampling budgets and the three-signal ensemble when calibration matters. Our results suggest that code UE
deserves code-specific design, and these three axes are a first instantiation rather than the last word.
2
Method
2.1
Problem Formulation
We study post-hoc uncertainty estimation for code generation. Given a natural-language prompt x describing a programming problem and a program y = (y1 , . . . , yT ) generated by an LLM πθ conditioned on x, an uncertainty estimator is a function U (x, y) ∈ R that scores how uncertain πθ is about its own output (larger = more uncertain). Groundtruth correctness is functional: y is correct iff it passes every test case in the problem’s official test suite (pass@1) (Chen et al., 2021), which is held out and never shown to πθ . The self-generated tests used by the functional signal in §2.4 are strictly separate. Since larger U indicates higher uncertainty, we report AUROC and PRAUC using −U as the score, with the binary correctness label as the positive class. 2.2
Lexical Uncertainty
Motivation. A single wrong token can break an entire program: one flipped operator, one off-byone index, one misspelled function name, and the program crashes or silently returns the wrong answer. Natural language is far more tolerant; a wrong word usually leaves the meaning recoverable from context. This asymmetry, which we call token fragility, implies that uncertainty in code is 2
sparse: correctness hinges on a small number of critical positions rather than being spread evenly across the sequence. Averaging entropy over the full output, the default in NL UE, dilutes these few decisive tokens. We therefore design our lexical signal to focus on the most uncertain positions rather than the full sequence.
MBPP pass@1 from 54% to 98% (Appendix D.2). Our algorithmic signal therefore elicits naturallanguage solution plans from πθ and scores their inter-sample agreement. Pipeline. For each prompt x, we elicit N naturallanguage solution plans {z (1) , . . . , z (N ) } directly from πθ , using a prompt that asks for 6–10 numbered reasoning steps and explicitly forbids code constructs such as for, while, or variable assignments (full prompt in Appendix A). The N plans are sampled independently with temperature τ ; each captures the model’s reasoning about the algorithmic solution, free of any particular implementation.
Token-Level Entropy. Given an input prompt x and a generated code sequence y = (y1 , y2 , . . . , yT ), we compute the token-level entropy at position t as X H(yt ) = − p(v | y<t , x) log p(v | y<t , x), v∈V
(1) where V is the vocabulary and p(v | y<t , x) is the model’s predicted probability for token v. To obtain a sequence-level uncertainty score U (y), we aggregate token-level entropies across the generated program, preserving information from a small number of potentially decisive high-uncertainty tokens.
Agreement Score. We measure inter-sample algorithmic agreement with step-aware ROUGE-L similarity, a structural matching metric tailored to the step-by-step nature of the elicited plans. The metric splits each plan into semantic steps, computes token-level ROUGE-L for each step pair, and aggregates scores through bidirectional max matching, capturing algorithmic consistency while ignoring superficial wording variations. For two plans z (i) , z (j) :
Top-K Max Entropy. A minimal sparse aggregator takes the single most uncertain token, U (y) = maxt H(yt ). Because a single entropy spike can be noisy or non-critical, we extend this to averaging the K largest token entropies: U (y) =
1 K
K X
H(yσ(i) ),
sim(z (i) , z (j) ) =
(2)
(3)
The algorithmic uncertainty for prompt x is the mean pairwise dissimilarity,
i=1
where σ sorts token positions in descending order of entropy. Max entropy is the special case K=1; Top-K is more robust to isolated spikes. K is the only hyperparameter, with sensitivity reported in §D.1. 2.3
2 · RLstep (z (i) , z (j) ) . len(z (i) ) + len(z (j) )
Ualgo (x) = 1 −
X 2 sim z (i) , z (j) , N (N − 1) i<j
(4) so low inter-plan agreement yields high uncertainty. The score is associated with the prompt x; when a per-candidate score is required, we assign Ualgo (x) to every candidate y drawn from x.
Algorithmic Uncertainty
Motivation. Measuring consistency directly on code can sometimes be hard. Two correct quicksorts can look entirely different syntactically; two near-identical implementations can encode different algorithms. Surface or embedding similarity (Kuhn et al., 2023; Malinin and Gales, 2021) therefore conflates implementation noise with genuine algorithmic disagreement. Pseudocode sidesteps both problems: it abstracts away naming and control-flow variants, so semantically equivalent solutions look alike, while still exposing genuine algorithmic differences. It also preserves the information needed for correctness: conditioning Qwen3-14B on ground-truth pseudo-code lifts
Why elicit plans directly, not abstract from sampled code. A natural alternative is to first sample N programs and then prompt πθ to abstract each into a pseudo-code summary. We prefer direct elicitation for two reasons. (i) The abstraction step is a second pass over a possibly wrong program, which compounds noise: an incorrect implementation often yields an incorrect abstraction that nonetheless looks plausible. (ii) Direct elicitation forces πθ to commit to an algorithmic plan at the natural-language level without an intermediate code-writing step that conflates implementation 3
and intent. Two other alternatives also fail. ASTbased equivalence on raw code is too strict: it treats semantically equivalent rewrites (loop ↔ comprehension) as disagreements and inflates uncertainty on correct samples. Execution-based equivalence requires running the programs and therefore collapses into the functional signal in §2.4. 2.4
2023) and related self-verification methods use selfgenerated tests to rerank or filter candidate programs, returning a preferred program; we instead use the pass pattern to score the model’s uncertainty about a given output, which is a calibration signal rather than a selection mechanism. The two uses are orthogonal: a CodeT-selected program still carries an uncertainty value under our scheme, and our score can gate or defer a CodeT decision.
Functional Uncertainty
Motivation. Code is executable: a candidate program can be run on inputs and its outputs compared to expected ones. The fraction of self-generated tests that a candidate passes is therefore a direct, behaviorally grounded signal of functional correctness, available in the code setting through execution at substantially lower cost than multi-sample consistency methods, and inaccessible to any NL UE method. Our functional signal turns this fraction into an uncertainty score by measuring how consistent the candidate’s runtime behavior is with the behavior the self-generated tests expect.
Self-test quality. Self-generated tests can be wrong, biased toward easy cases, or fail to exercise edge conditions. We quantify the gap to the official test suite in §D.2 by varying M and comparing against an oracle upper bound, and discuss it in Limitations. 2.5
The three signals, namely Ulex (Top-K token entropy, §2.2), Ualgo (pseudo-code consistency, §2.3), and Ufunc (behavioral consistency, §2.4), live on different scales and have different empirical ranges. Before combining them, we map each score to its empirical rank on the evaluation set and rescale to [0, 1]; let Ũ• denote the rank-normalized version of U• . The ensemble is a simple weighted sum,
Pipeline. For each prompt x, we (i) prompt πθ to self-generate M test cases T (x) = {t1 , . . . , tM } (prompt in Appendix A); (ii) take the candidate program y produced for x (the greedy generation, in our default setup); and (iii) execute y against each test in a sandboxed interpreter, treating compile errors, runtime errors, and timeouts as failures.
Uens (x, y) = αlex Ũlex (x, y) + αalgo Ũalgo (x) + αfunc Ũfunc (x, y),
Consistency Score. Let Pj (y) = ⊮[y passes tj ]. We define the candidate’s behavioral consistency with its self-tests as the fraction of tests passed, 1 X Pj (y), M
(5)
j=1
and the functional uncertainty as the complementary disagreement, Ufunc x, y
= 1 − C(y).
(7)
with non-negative weights summing to one. Details of the choice of ensemble weights are in Appendix C. We deliberately keep the combiner simple. The ensemble is not the contribution of the paper; its role is to demonstrate that the three axes are complementary, a claim we substantiate directly in §5 via three analyses.
M
C(y) =
Ensemble
3
(6)
Experimental Setup
Datasets. We evaluate on four widely used Python code generation benchmarks that together span a broad difficulty range. APPS (Hendrycks et al., 2021) provides programming problems collected from competitive-coding platforms; we use its Introductory (N = 1000) and Interview (N = 1000) subsets, which differ substantially in algorithmic depth and so let us probe whether each uncertainty signal degrades gracefully with problem difficulty. HumanEval (Chen et al., 2021) (N = 164) and MBPP (Austin et al., 2021) (N =
A candidate that disagrees with most of its own problem’s self-tests is flagged as uncertain. This formulation operates on a single candidate; aggregations exploiting multiple candidates’ joint pass/fail patterns (pairwise Hamming agreement, pertest Bernoulli entropy) are an alternative we leave to future work. Differentiation from selection-based methods. The functional signal is closely related to testcase-based program selection. CodeT (Chen et al., 4
500) are community-standard benchmarks of short, function-level problems with hidden unit tests. HumanEval and MBPP are now close to saturated for strong code LLMs in pass@1; APPS-Interview leaves clear headroom and provides a harder distribution on which calibration matters most.
functional signal we score the greedy candidate against M =10 self-generated tests with a per-test execution timeout of 4 s; compile errors, runtime errors, and timeouts are treated as failures. Ensemble weights are fixed at (αlex , αfunc , αalgo ) = (0.2, 0.4, 0.4). Full prompt templates and additional decoding settings are in Appendix B.
Models. We evaluate on five open-source code LLMs spanning 14B–32B parameters and a mix of general and code-specialized models: Qwen3-14B and Qwen3-32B (Yang et al., 2025), the code-specialized Qwen3-Coder-30B-A3BInstruct (Yang et al., 2025), denoted as Qwen3Coder-30B, Mistral-Devstral-Small-2505 (Rastogi et al., 2025), denoted as Devstral-Small-2505, and DeepSeek-Coder-V2-Lite-Instruct (Zhu et al., 2024), denoted as Deepseek-Coder-V2. We report three representative models (Qwen3-14B, DeepSeek-Coder-V2, and Devstral-Small-2505) in Table 1 and defer the full per-model breakdown to Appendix D.
4
Main Results
Table 1 reports AUROC and PRAUC on all four benchmarks for Qwen3-14B, DeepSeek-Coder-V2, and Devstral-Small-2505; Results on more models are availbale in Appendix D. Each axis captures complementary uncertainty. Top-5 token entropy (lexical) excels on algorithmically demanding APPS subsets (0.813 AUROC on Intro for Qwen3-14B), matching the strongest multi-pass NL baseline (Consistency-vr: 0.728 average) at over 3× lower cost. Generated Tests (functional) dominates on executable benchmarks, achieving 0.822 on HumanEval and 0.730 on MBPP. Pseudo-code consistency (algorithmic) provides orthogonal signal, particularly strong on Devstral-Small-2505 (0.716 average AUROC).
Baselines. We compare our method against two families. Single-pass methods score one greedy generation: Mean Entropy, which aggregate raw entropy without Top-K filtering. Multi-pass methods constitute the current state of the art for UE in both code and long-form text generation: Consistency (Basic) and Consistency (VR) (Huang et al., 2025), both using pairwise CodeBLEU (Malinin and Gales, 2021); and Symbolic Clustering (Sharma and David, 2025); Within our own framework, Max Entropy serves as natural K=1 ablations of the lexical signal.
The ensemble yields substantial gains across all models. The weighted ensemble (0.2/0.4/0.4) raises average AUROC to 0.800 for Qwen3-14B (+7.2 over best single signal), 0.770 for DeepSeekCoder-V2 (+1.9), and 0.791 for Devstral-Small2505 (+2.7), confirming complementarity. The ensemble is the top method on every benchmark in Table 1, with particularly strong gains on HumanEval (0.852 AUROC, 0.983 PRAUC for Qwen3-14B).
Metrics. We measure discrimination with AUROC and ranking quality with PRAUC, with the positive class set to correct and −U (i.e., confidence) as the ranking score. Correctness is defined functionally: the label y=1 iff the candidate program passes the problem’s official hidden test suite (not the self-generated tests of §2.4). This distinction is critical, since otherwise the functional signal would be trivially perfect by construction. We do not report Brier Score or ECE: post-hoc rank-normalized scores have no meaningful absolute calibration, and AUROC/PRAUC suffice to assess the underlying signal.
5
Analysis
Orthogonality of the Three Signals. While Table 1 demonstrates that each axis outperforms NLderived baselines individually and that the ensemble yields further gains, neither observation directly establishes that the three signals capture genuinely distinct information. We provide three complementary lines of evidence. First, pairwise correlations (Table 2) are low, with Pearson and Spearman coefficients ranging 0.10–0.22; the lexical–functional pair is most decoupled (r=0.183, ρ=0.221), consistent with targeting structurally distinct failure modes (sparse high-entropy token decisions vs. end-to-end execution). Second, drop-one ablation (Table 3) reveals non-trivial degradation when removing any signal: dropping the functional axis
Hyperparameters. For the lexical signal we use K=5 (Top-K token entropy) on a single greedy generation (τ =0); sensitivity to K is reported in §D.1. For the algorithmic signal we sample N =10 pseudo-code plans at temperature τ =0.8. For the 5
APPS Intro
Method
APPS Interview
HumanEval
MBPP
Average
AUROC PRAUC AUROC PRAUC AUROC PRAUC AUROC PRAUC AUROC PRAUC Qwen3-14B NL-derived baselines Mean Entropy Consistency (BLEU) Consistency (VR) Symb. Clustering Three axes (ours) Top-5 Entropy (LEX) Pseudo Consistency (ALGO) Generated Tests (FUNC) Ensemble (ours) Top-5 + Tests + Pseudo
.694 .742 .776 .764
.700 .776 .798 .813
.691 .723 .748 .690
.414 .495 .525 .522
.715 .694 .741 .673
.941 .945 .955 .943
.526 .640 .648 .661
.565 .673 .684 .691
.657 .700 .728 .697
.655 .722 .741 .742
.813 .713 .745
.828 .833 .787
.798 .604 .755
.611 .588 .549
.729 .718 .822
.952 .947 .958
.570 .615 .730
.616 .650 .702
.728 .662 .763
.752 .755 .749
.792
.870
.810
.725
.852
.983
.746
.761
.800
.835
DeepSeek-Coder-V2 NL-derived baselines Mean Entropy Consistency (BLEU) Consistency (VR) Symb. Clustering Three axes (ours) Top-5 Entropy (LEX) Pseudo Consistency (ALGO) Generated Tests (FUNC) Ensemble (ours) Top-5 + Tests + Pseudo
.483 .721 .759 .566
.532 .767 .797 .643
.528 .703 .747 .473
.277 .470 .522 .277
.376 .586 .635 .646
.798 .850 .877 .895
.544 .596 .616 .565
.611 .641 .639 .639
.483 .652 .689 .563
.555 .682 .709 .614
.637 .691 .708
.668 .802 .763
.634 .653 .692
.368 .500 .501
.466 .759 .861
.827 .832 .952
.616 .722 .741
.653 .636 .736
.588 .706 .751
.629 .693 .738
.749
.814
.747
.558
.842
.957
.740
.755
.770
.771
Devstral-Small-2505 NL-derived baselines Mean Entropy Consistency (BLEU) Consistency (VR) Symb. Clustering Three axes (ours) Top-5 Entropy (LEX) Pseudo Consistency (ALGO) Generated Tests (FUNC) Ensemble (ours) Top-5 + Tests + Pseudo
.491 .614 .745 .484
.461 .624 .737 .506
.558 .609 .675 .461
.189 .246 .281 .190
.612 .616 .644 .668
.876 .885 .875 .893
.528 .597 .611 .514
.558 .621 .642 .575
.547 .609 .669 .532
.521 .594 .634 .541
.701 .803 .747
.651 .704 .748
.627 .646 .712
.239 .433 .392
.749 .768 .819
.934 .939 .940
.614 .647 .776
.653 .620 .735
.673 .716 .764
.619 .674 .704
.752
.739
.734
.449
.886
.990
.791
.785
.791
.741
Table 1: Main results on Qwen3-14B, DeepSeek-Coder-V2 and Devstral-Small-2505: AUROC (AUC) and PRAUC across four benchmarks. Per-model tables are in Appendix D. Bold indicates best performance; underline indicates second-best performance.
Pair
Pearson r
Spearman ρ
Pseudo vs Test Pseudo vs Entropy Test vs Entropy
0.0997 0.1149 0.1830
0.1085 0.1154 0.2212
Table 2: Pairwise correlations among the three signals on MBPP with Qwen3-Coder-30B. Low correlations indicate largely non-redundant information.
Variant
AUROC
PRAUC
Full (P+T+E) Drop P (T+E) Drop T (P+E) Drop E (P+T)
0.7478 0.7392 0.6172 0.7457
0.7672 0.7416 0.6758 0.7574
Table 3: Drop-one ablation on MBPP with Qwen3Coder-30B. Removing any signal degrades performance, with the test signal (T) contributing most.
reduces AUROC by 13.1 points, while omitting algorithmic or lexical axes reduces AUROC by 0.9 and 0.2 points respectively, confirming no signal is subsumed by the other two. Third, per-sample disagreement analysis (Figure 2) shows each axis is most reliable on distinct error classes: lexical on
single-token failures, algorithmic when samples implement different solution strategies, and functional when code is syntactically valid but semantically incorrect.
6
Number of samples
Average Time per Problem (seconds)
Correct Incorrect
20 15 10 5 0
0.0
0.1
0.2
0.3
0.4
Std of percent ranks across P/T/E axes
0.5
8 7
7.10s
6.55s
6 5 4
3.06s
3
2.29s
2 1 0
VR (Code)
Pseudo (Pseudo-code)
Top5-ENT (Entropy)
Testcases
Uncertainty Estimation Method
(a) Disagreement-score distributions for correct vs. incorrect.
Number of samples
300
Figure 3: Computational efficiency comparison of uncertainty estimation methods. Average time per problem on HumanEval (164 problems) with Qwen3-14B model.
Correct Incorrect
250 200
Token Scope
AUROC↑
PRAUC↑
150
All Tokens Code Only Comment Only
0.728 0.716 0.375
0.752 0.726 0.471
100 50 0
Pseudo
Test
Winner axis on each sample
Table 4: Top-5 max entropy applied to different token scopes (Qwen3-14B, averaged across datasets).
Entropy
is acceptable.
(b) Counts of confidence signal per sample, by correctness.
Code vs. Comment Entropy. Code and comments serve fundamentally different roles in a program. Table 4 decomposes Top-5 max entropy by token scope. Code-only entropy achieves competitive performance (AUROC 0.716), confirming that executable tokens carry the primary uncertainty signal. Comment-only entropy, by contrast, performs worse than random (AUROC 0.375), indicating that the model’s uncertainty about natural-language explanations anti-correlates with functional correctness. Yet all-token entropy (0.728) modestly outperforms code-only (0.716), suggesting that while comment tokens are individually unreliable, their entropy distribution interacts with code-token entropy to sharpen the overall rank ordering. This asymmetry is invisible to methods that aggregate uncertainty uniformly over the full sequence and would be missed by NL-derived baselines. Practitioners should retain comment tokens when computing entropy-based uncertainty but avoid relying on verbalized confidence as a proxy for code correctness. In summary, executable code tokens are the dominant source of uncertainty signal, yet naively discarding comments sacrifices a modest but consistent gain.
Figure 2: Per-sample disagreement analysis on Qwen3Coder-30B on MBPP.
Cost–Performance Pareto. Figure 3 reports wall-clock time per problem for entropy- and consistency-based methods on HumanEval with Qwen3-14B. Top-K entropy requires only 2.29 seconds per problem, approximately 3.1× faster than sampling-based consistency via code (VR: 7.10s) and 2.9× faster than pseudo-code consistency (6.55s). Generated test cases take 3.06 seconds, slightly slower than entropy but still 2.3× faster than VR. Combined with Table 1, this places TopK entropy in the lower-left of the cost–AUROC plane (fast, competitive accuracy), while the full three-axis ensemble achieves the strongest overall performance by aggregating lexical, algorithmic, and functional signals. The full three-signal ensemble adds pseudo-code consistency, increasing runtime by 6.55s but yielding only marginal AUROC gains on most benchmarks. We therefore recommend Top-K entropy as the default for latencysensitive settings (e.g., in-IDE autocomplete), and the full three-axis ensemble when calibration matters more than speed and the added cost of aggregating lexical, algorithmic, and functional signals
Generalization to Other Languages. We evaluate whether the lexical and consistency-based 7
Method
C++ Java
Go
Mean Entropy Max Entropy Mean Probability Max Probability Consistency (BLEU) Consistency (VR) Symb. Clustering Top-5 Entropy (ours) Pseudo Consistency(ours) Generated Tests(ours) Top-5 + Tests + Pseudo(ours)
.543 .624 .510 .582 .649 .679 .660 .612 .523 .502 .588 .513
tropy as a useful signal for confidence or reasoning quality in NL and code tasks (Wang et al., 2025; Li et al., 2026; Cooper and Scholak, 2024; Zeng et al., 2025; Shi et al., 2025a). None of these methods is code-aware: they view programs merely as token sequences, relying on uniform sequence uncertainty and surface clustering to approximate semantic consistency, without accounting for executability. Our framework keeps the multi-sampleagreement intuition of consistency-based methods but instantiates it at code-specific abstraction levels, and motivates Top-K aggregation specifically from token fragility.
JS
.581 .682 .651 .583 .572 .665 .650 .597 .648 .690 .673 .625
Table 5: Cross-language AUROC on Qwen3-Coder-30B (HumanEval-X). Top-5 max entropy is consistently the strongest signal among methods that transfer to nonPython settings.
Uncertainty and self-verification for code. Direct work on code UE is sparse. Sharma and David (2025) adapt entropy- and mutual-informationbased UE to code with a symbolic-execution semantic-equivalence check; LUQ (Zhang et al., 2024) is a sampling-based UE method for longform generation; structural-entropy (Song et al., 2025) and complexity-feedback (Sepidband et al., 2025) derive signals from candidate programs themselves. A parallel line uses self-generated tests to select or repair programs rather than score uncertainty: CodeT (Chen et al., 2023) ranks candidates by self-test pass count, while self-debug (Chen et al., 2024; Shi et al., 2025b; Li et al., 2025; Chen et al., 2025) and self-edit (Zhang et al., 2023)iteratively repair programs from execution feedback. Each existing UE method commits to a single equivalence definition, covering implementation-level symbolic execution, surface sampling and natural language-derived atomic claim decomposition while self-verification outputs selected programs instead of calibration cues. We contend no single equivalence standard suits code, defining lexical, algorithmic and functional disagreement as three orthogonal dimensions and tests their orthogonality (§5); the functional axis can be layered on top of CodeT-style selection without conflict.
axes transfer to non-Python languages by testing Qwen3-Coder-30B on HumanEval-X (C++, Java, Go, JavaScript). The functional axis is omitted due to the complexity of a uniform sandbox across four languages. Table 5 shows that Top-5 max entropy is the strongest signal on Java, Go and JS, tied on C++, outperforming both single-pass baselines (Max Entropy, Line Max) and the multi-pass consistency baseline (VR). This replicates the Python finding and confirms that the lexical uncertainty signal generalizes across programming languages. Practical guidance. Top-K token entropy is the default for latency-sensitive settings (e.g., inIDE autocomplete): it is single-pass and competitive with sampling-based methods. The full three-signal ensemble (Top-K + behavioral consistency + pseudo-code consistency) provides the strongest average performance when calibration matters more than latency and the added cost is acceptable. The algorithmic axis is best used as a complement, not in isolation.
6
Related Work
Uncertainty estimation for natural language generation. Confidence and uncertainty estimation in LLMs has been studied along several axes: post-hoc calibration (Guo et al., 2017; Zadrozny and Elkan, 2002), sampling-based consistency and semantic clustering (Kuhn et al., 2023; Malinin and Gales, 2021; Lin et al., 2024), verbalized confidence and P(True) prompts (Tian et al., 2023; Xiong et al., 2024; Kadavath et al., 2022), and long-form decomposition into atomic claims (Zhang et al., 2025, 2024); surveys cover the landscape (Geng et al., 2024; Xiong et al., 2024). A complementary thread highlights token-level en-
7
Conclusion
We introduce a three-axis framework for code uncertainty estimation that maps one-to-one onto three properties distinguishing code from natural language: token fragility, two-level structure, and executability. Across four benchmarks and five code LLMs, the three axes provide complementary signals for code uncertainty estimation. In our main Qwen3-14B setting, Top-K entropy alone matches the strongest multi-pass NL baseline at 8
over 3× lower cost, while across models it remains a competitive low-cost estimator. The full threeaxis ensemble further improves average AUROC, reaching 0.800 on Qwen3-14B and 0.776 across all five models. Our results suggest that as code-LLM deployment matures, calibration, not further capability scaling, becomes the bottleneck, and codespecific signals such as type checks, static analyses, and runtime traces should expand the framework rather than be forced into NL-style estimators.
The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 39 others. 2021. Evaluating large language models trained on code. Preprint, arXiv:2107.03374. Silin Chen, Shaoxin Lin, Xiaodong Gu, Yuling Shi, Heng Lian, Longfei Yun, Dong Chen, Weiguo Sun, Lin Cao, and Qianxiang Wang. 2025. Swe-exp: Experience-driven software issue resolution. arXiv preprint arXiv:2507.23361.
Limitations Our study focuses on Python code generation with hidden test suites; preliminary cross-language results (Section 5) suggest the lexical and consistency axes transfer, and extending the framework to open-ended code tasks such as refactoring and multi-file edits is a natural next step. The functional axis assumes an executable environment, which is standard in code-generation benchmarks but not universal; the lexical and algorithmic axes remain applicable when execution is unavailable. We evaluate discrimination and ranking quality (AUROC, PRAUC) and leave the study of downstream utility (selective generation, human-in-the-loop review, agentic deferral) to future work.
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2024. Teaching large language models to self-debug. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. Nathan Cooper and Torsten Scholak. 2024. Perplexed: Understanding when large language models are confused. Preprint, arXiv:2404.06634. Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. 2024. A survey of confidence estimation and calibration in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pages 6577–6595. Association for Computational Linguistics.
AI Usage Disclosure The authors used ChatGPT to refine the manuscript’s grammar. All AI-assisted text was reviewed and revised by the authors, who take full responsibility for the final version.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, Proceedings of Machine Learning Research, pages 1321–1330. PMLR.
Ethical Statement This study uses only publicly available datasets (APPS, HumanEval, MBPP), obtained and cited in compliance with their respective licenses. No private data, human subjects, or sensitive information are involved, and the work follows standard academic integrity norms, with research-derived data not used outside research contexts.
References
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. Measuring coding challenge competence with APPS. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. Program synthesis with large language models. Preprint, arXiv:2108.07732.
Yuheng Huang, Jiayang Song, Zhijie Wang, Shengming Zhao, Huaming Chen, Felix Juefei-Xu, and Lei Ma. 2025. Look before you leap: An exploratory study of uncertainty analysis for large language models. IEEE Trans. Software Eng., 51(2):413–429.
Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2023. Codet: Code generation with generated tests. In
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli
9
Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, and 17 others. 2022. Language models (mostly) know what they know. Preprint, arXiv:2207.05221.
Yuling Shi, Songsong Wang, Chengcheng Wan, Min Wang, and Xiaodong Gu. 2025b. From code to correctness: Closing the last mile of code generation with hierarchical debugging. Preprint, arXiv:2410.01215.
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
Yuling Shi, Hongyu Zhang, Chengcheng Wan, and Xiaodong Gu. 2025c. Between lines of code: Unraveling the distinct patterns of machine and human programmers. In 47th IEEE/ACM International Conference on Software Engineering, ICSE 2025, Ottawa, ON, Canada, April 26 - May 6, 2025, pages 1628– 1639. IEEE.
Han Li, Yuling Shi, Shaoxin Lin, Xiaodong Gu, Heng Lian, Xin Wang, Yantao Jia, Tao Huang, and Qianxiang Wang. 2025. Swe-debate: Competitive multiagent debate for software issue resolution. arXiv preprint arXiv:2507.23348.
Yewei Song, Tiezhu Sun, Xunzhu Tang, Prateek Rajput, Tegawendé F. Bissyandé, and Jacques Klein. 2025. Measuring LLM code generation stability via structural entropy. In 40th IEEE/ACM International Conference on Automated Software Engineering, ASE 2025, Seoul, Korea, Republic of, November 16-20, 2025, pages 3922–3926. IEEE.
Xianzhi Li, Ethan Callanan, Abdellah Ghassel, and Xiaodan Zhu. 2026. Entropy-gated branching for efficient test-time reasoning. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2026 - Volume 1: Long Papers, Rabat, Morocco, March 24-29, 2026, pages 5054–5069. Association for Computational Linguistics.
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D. Manning. 2023. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 5433–5442. Association for Computational Linguistics.
Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2024. Generating with confidence: Uncertainty quantification for black-box large language models. Trans. Mach. Learn. Res., 2024. Andrey Malinin and Mark J. F. Gales. 2021. Uncertainty estimation in autoregressive structured prediction. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. 2025. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning. Preprint, arXiv:2506.01939.
Abhinav Rastogi, Adam Yang, Albert Q. Jiang, Alexander H. Liu, Alexandre Sablayrolles, Amélie Héliou, Amélie Martin, Anmol Agarwal, Andy Ehrenberg, Andy Lo, Antoine Roux, Arthur Darcet, Arthur Mensch, Baptiste Bout, Baptiste Rozière, Baudouin De Monicault, Chris Bamford, Christian Wallenwein, Christophe Renaudin, and 84 others. 2025. Devstral: Fine-tuning language models for coding agent applications. Preprint, arXiv:2509.25193.
Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2024. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.
Melika Sepidband, Hamed Taherkhani, Song Wang, and Hadi Hemmati. 2025. Enhancing llm-based code generation with complexity metrics: A feedbackdriven approach. In 49th IEEE Annual Computers, Software, and Applications Conference, COMPSAC 2025, Toronto, ON, Canada, July 8-11, 2025, pages 1416–1426. IEEE.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
Arindam Sharma and Cristina David. 2025. Assessing correctness in llm-based code generation via uncertainty estimation. Preprint, arXiv:2502.11620.
Bianca Zadrozny and Charles Elkan. 2002. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’02, page 694–699, New York, NY, USA. Association for Computing Machinery.
Yuling Shi, Yichun Qian, Hongyu Zhang, Beijun Shen, and Xiaodong Gu. 2025a. Longcodezip: Compress long context for code language models. arXiv preprint arXiv:2510.00446.
10
where {problem_description} contains the function signature and docstring for HumanEval/MBPP, or the full problem statement with optional starter code for APPS. Code generation uses greedy decoding (temperature=0.0, top-p=0.95, max_tokens=1024).
Wenhao Zeng, Yaoning Wang, Chao Hu, Yuling Shi, Chengcheng Wan, Hongyu Zhang, and Xiaodong Gu. 2025. Pruning the unsurprising: Efficient code reasoning via first-token surprisal. arXiv preprint arXiv:2508.05988. Caiqi Zhang, Fangyu Liu, Marco Basaldella, and Nigel Collier. 2024. LUQ: Long-text uncertainty quantification for LLMs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5244–5262, Miami, Florida, USA. Association for Computational Linguistics.
A.2
For pseudo-code consistency estimation, we generate multiple high-level solution plans without writing actual code:
Caiqi Zhang, Ruihan Yang, Zhisong Zhang, Xinting Huang, Sen Yang, Dong Yu, and Nigel Collier. 2025. Atomic calibration of llms in long-form generations. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, IJCNLP-AACL 2025, Mumbai, India, December 20-24, 2025, pages 148–169. The Asian Federation of Natural Language Processing and The Association for Computational Linguistics.
Pseudo-Code Generation Prompt <user> Read the problem and describe the solution logic in a step,→ by-step plan. Problem: {problem_description} Write a solution plan with 6-10 numbered steps that: - Describes the core algorithm logic and reasoning - Explains what needs to be done and why - Uses natural language like "examine each item", "keep ,→ track of", "compare values" - Avoids programming constructs (no "for", "while", "if,→ else", variable assignments) - Focuses on the logical flow: "first do X, then check Y, ,→ finally return Z"
Kechi Zhang, Zhuo Li, Jia Li, Ge Li, and Zhi Jin. 2023. Self-edit: Fault-aware code editor for code generation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 769–787. Association for Computational Linguistics.
Output the plan directly. </user> <assistant> <think></think> Solution plan:
Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y. Wu, Yukun Li, Huazuo Gao, Shirong Ma, and 1 others. 2024. DeepSeekCoder-V2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406.11931.
A
We generate 10 pseudo-code plans per problem using temperature sampling (Temperature=0.8, Top-p=0.95, Max_tokens=2048) and compute pairwise step-aware ROUGE-L similarity to measure reasoning consistency.
Prompts
A.3
This section provides the exact prompts used in our experiments for code generation, pseudo-code generation, and test case generation. A.1
Pseudo-Code Generation Prompt
Test Case Generation Prompt
Test Case Generation Prompt <user> Generate {num_test_cases} DIFFERENT test cases for this ,→ function. DO NOT implement the function.
Code Generation Prompt
We use a unified prompt template across all benchmarks (HumanEval, MBPP, and APPS):
Function Specification: {problem_description} Requirements: 1. Generate exactly {num_test_cases} diverse test cases as ,→ valid Python assert statements 2. Use the exact function name and parameters from the ,→ specification 3. Cover different scenarios: edge cases, normal cases, ,→ boundary conditions 4. Output ONLY valid JSON array with {num_test_cases} ,→ assert statements
Code Generation prompt <user> Complete the following Python Code: {problem_description} Output only the complete code with brief comments, when you ,→ output ```, the code should be complete and executable and you should ,→ stop immediately. </user> <assistant> <think></think> Here are the complete codes for this problem: ```python
JSON format: [ "assert function_name(args1) == expected1", "assert function_name(args2) == expected2", ... ]
11
APPS_INTRODUCTORY
APPS_INTERVIEW
HUMANEVAL
0.80 0.774 0.781
AUROC
0.75
0.802
0.810 0.781
0.809
0.809
0.809
0.808
0.807
0.760
0.763
0.762
0.762
0.761
0.761
0.760
0.644
0.644
0.650
0.652
0.613
0.613
0.613
0.613
top-9
top-11
top-15
top-20
0.738 0.686
0.70
0.657
0.65
We generate test cases to evaluate code correctness without executing against ground-truth tests: We generate 10 test cases per problem using sampling decoding (Temperature=0.8, Top-p=0.95, Max_tokens=1024). For APPS problems with standard I/O format, we adapt the output to {"input": "...", "output": "..."} pairs instead of assert statements.
B
0.60
0.598
top-1
top-3
0.613
top-5
top-7
Top-k
Qwen3-14B block reproduces Table 1 for reference.
Hyperparameters and Experiment Details
Across all five models, ensemble consistently dominates: The Top-5 + Tests + Pseudo ensemble achieves the best (or tied-best) performance on every {model, dataset} cell tested, demonstrating that the three axes—lexical (LEX), algorithmic (ALGO), and functional (FUNC)—capture complementary signals. While individual methods show dataset-specific strengths (Top-5 Entropy excels on APPS, Generated Tests on HumanEval/MBPP), the ensemble robustly combines their advantages: it matches or exceeds the best single method on each benchmark, with gains of up to +8.7 PRAUC points (Qwen3-14B on APPS Interview) over the strongest baseline. The relative contribution of each axis varies by model—stronger code models (Qwen3-Coder-30B, Devstral-Small-2505) benefit more from lexical diversity, while smaller models (DeepSeek-Coder-V2) gain more from functional verification—but the ensemble remains the most reliable estimator across all settings.
Analysis of Ensemble
To determine the optimal ensemble weights (αlex , αalgo , and αfunc ), we perform a simple grid search on a small held-out validation set partitioned from the training data, rather than tuning directly on the evaluation benchmarks. We evaluate the combination of weights with a step size of 0.1 and select the configuration that maximizes the overall AUROC score. The performance remains consistently high across a broad range of weight combinations, demonstrating that our ensemble method is robust to hyperparameter choices and does not suffer from severe overfitting.
D
0.634 0.601
Figure 4: AUROC of Top-K entropy as a function of K, averaged across models and benchmarks. Performance improves from K=1 to K=5, then plateaus.
In our experiments, we use 10 samples per instance for consistency calculation and clustering. As depicted in Table 6, for the LLM, we set the temperature to 0.8 during consistency calculation and to 0 when generating outputs for entropy computation. The temperature is set to 1 when generating perturbed results for entropy analysis. For Top-K max entropy we use K=5 as the default. The ensemble weights used in the main results are αlexical =0.2, αfunctional =0.4, αalgorithmic =0.4 after rank-normalization.
C
MBPP
0.85
Output only the JSON, when you output ```, the JSON should ,→ be complete and you should stop immediately. </user> <assistant> <think></think> Here are the test cases in JSON format: ```json
D.1
Method-specific Ablations
Top-K entropy: a small K captures most of the signal; performance plateaus around K=5 (§D.1). Figure 4 shows how AUROC varies with K. Performance improves steadily from K=1 (max entropy) to K=5 and then plateaus, matching the token fragility hypothesis: a few high-entropy tokens carry the most informative uncertainty signal, and as K grows, lower-entropy boilerplate tokens dilute it. We use K=5 as the default.
Per-Model, Per-Dataset Results
Table 9 and Table 10 report AUROC and PRAUC for all five models across all four benchmarks, including single-pass entropy methods, multi-pass NL-derived consistency baselines, our functional axis (Generated Tests), and both ensembles. The 12
Hyperparameter
Value
Descriptions
Temperatures τ when generating outputs Temperatures τ for entropy analysis Temperatures τ during consistent calculation
0 1 0.8
Temperatures τ when generating pseudo codes
0.8
Greedy decoding for deterministic final outputs Sampling temperature for entropy distribution analysis Sampling temperature for generating diverse code samples in VR and algorithmic consistency Sampling temperature for diverse pseudo-code generation Sampling temperature for diverse test case generation Number of samples per problem for consistency metrics Number of test cases per code for functional uncertainty Number of highest-entropy tokens for Top5_ent metric Combination weights for lexical, algorithmic, and functional uncertainty Nucleus sampling threshold for all diverse generation Random seed for reproducibility
Temperatures τ when generating testcases Sample Number N Test Case Number M Top-K K Ensemble Weights αlex , αalgo , αfunc
0.8 10 10 5 0.2, 0.4, 0.4
Top_P Seed
0.95 42
Table 6: Hyperparameter settings and configurations for uncertainty estimation and output generation. APPS Intro
Model Qwen3-14B Qwen3-32B Qwen3-Coder-30B DeepSeek-Coder-V2 Devstral-Small-2505
APPS Interview
HumanEval
MBPP
Average
Acc
AUROC
PRAUC
Acc
AUROC
PRAUC
Acc
AUROC
PRAUC
Acc
AUROC
PRAUC
Acc
AUROC
PRAUC
0.569 0.590 0.606 0.549 0.479
0.745 0.742 0.749 0.708 0.747
0.787 0.804 0.809 0.763 0.748
0.282 0.281 0.341 0.261 0.161
0.755 0.770 0.738 0.692 0.712
0.549 0.571 0.576 0.501 0.392
0.875 0.869 0.950 0.825 0.799
0.822 0.756 0.734 0.861 0.819
0.958 0.938 0.973 0.952 0.940
0.540 0.580 0.556 0.564 0.530
0.730 0.711 0.731 0.741 0.776
0.702 0.724 0.722 0.736 0.735
0.567 0.580 0.613 0.550 0.492
0.763 0.745 0.738 0.751 0.764
0.749 0.759 0.770 0.738 0.704
Table 7: Problem-level evaluation of LLM-generated test cases as a proxy for code correctness.
implementation details while preserving the algorithmic content needed for correctness, justifying it as the abstraction level for our algorithmic signal. Test-case consistency: discrimination improves with more self-generated tests, but plateaus below the oracle (§D.2). This suggests room for test-generation strategies optimized for uncertainty estimation (coverage- or boundary-aware prompts) rather than for program selection.
E Figure 5: AUROC and PRAUC of testcases as a function of the testcase number.Performance is the best when the number is 10.
D.2
Compute Budget
All experiments were conducted on a single NVIDIA A100-SXM4-80GB GPU. We report the GPU hours required for each uncertainty estimation method on the HumanEval benchmark (164 problems) using the Qwen3-14B model (14B parameters).
Pseudo-Code Preserves Algorithmic Information
We validate that pseudo-code retains the information needed for correct implementation. On 100 MBPP problems, we provide [MODEL] with the reference solution and prompt it to produce pseudocode, then ask Qwen3-14B to implement each problem conditioned on the pseudo-code. Pass@1 reaches 98%, compared to 54% for Qwen3-14B’s unconditional generation on MBPP (Table 8). This confirms that pseudo-code abstracts away surface
Per-experiment GPU hours: • Consistency(VR): 0.32 hours (1161s generation + 4s VR calculation + 2s metrics execution = 1167s total) • Pseudo-code Consistency: 0.30 hours (1067s generation + 8s VR calculation + 2s metrics execution = 1077s total) 13
APPS Introductory APPS Interview HumanEval MBPP
Model Qwen3-Coder-30B Qwen3-14B Qwen3-32B DeepSeek-Coder-V2 Devstral-Small-2505
61.20% 56.60% 58.90% 53.40% 48.00%
31.50% 28.60% 27.30% 26.00% 16.00%
94.51% 88.41% 87.80% 81.10% 80.49%
56.00% 54.20% 58.00% 57.00% 52.00%
Table 8: Pass rate of generated codes.
• Top5 Token Entropy: 0.10 hours (210s generation + 163s entropy calculation + 2s metrics execution = 375s total) • Behavioral Consistency: 0.14 hours (231s generation + 269s test generation + 2s execution = 502s total)
F
Pass Rates of Generated Results
Table 8 presents the pass@1 rates of our code solutions generated by the five evaluated LLMs across all four benchmarks. Model performance varies across benchmarks and these results validate the reliability of our dataset used in the uncertainty estimation.
14
Method
APPS Intro
APPS Interview
HumanEval
MBPP
Average
AUROC PRAUC AUROC PRAUC AUROC PRAUC AUROC PRAUC AUROC PRAUC Qwen3-14B NL-derived baselines Mean Entropy Max Entropy Mean Probability Max Probability Consistency (BLEU) Consistency (VR) Symb. Clustering Three axes (ours) Top-5 Entropy (LEX) Pseudo Consistency (ALGO) Generated Tests (FUNC) Ensemble (ours) Top-5 + Tests + Pseudo DeepSeek-Coder-V2 NL-derived baselines Mean Entropy Max Entropy Mean Probability Max Probability Consistency (BLEU) Consistency (VR) Symb. Clustering Three axes (ours) Top-5 Entropy (LEX) Pseudo Consistency (ALGO) Generated Tests (FUNC) Ensemble (ours) Top-5 + Tests + Pseudo Devstral-Small-2505 NL-derived baselines Mean Entropy Max Entropy Mean Probability Max Probability Consistency (BLEU) Consistency (VR) Symb. Clustering Three axes (ours) Top-5 Entropy (LEX) Pseudo Consistency (ALGO) Generated Tests (FUNC) Ensemble (ours) Top-5 + Tests + Pseudo
.694 .724 .716 .720 .742 .776 .764
.700 .767 .751 .718 .776 .798 .813
.691 .672 .582 .625 .723 .748 .690
.414 .411 .408 .513 .495 .525 .522
.715 .601 .619 .630 .694 .741 .673
.941 .898 .902 .935 .945 .955 .943
.526 .532 .518 .521 .640 .648 .661
.565 .569 .555 .564 .673 .684 .691
.657 .632 .609 .624 .700 .728 .697
.655 .661 .654 .683 .722 .741 .742
.813 .713 .745
.828 .833 .787
.798 .604 .755
.611 .588 .549
.729 .718 .822
.952 .947 .958
.570 .615 .730
.616 .650 .702
.728 .662 .763
.752 .755 .749
.792
.870
.810
.725
.852
.983
.746
.761
.800
.835
.483 .503 .499 .510 .721 .759 .566
.532 .534 .528 .530 .767 .797 .643
.528 .652 .519 .546 .703 .747 .473
.277 .349 .318 .341 .470 .522 .277
.376 .412 .393 .425 .586 .635 .646
.798 .826 .767 .788 .850 .877 .895
.544 .601 .596 .617 .596 .616 .565
.611 .620 .612 .634 .641 .639 .639
.483 .542 .502 .525 .652 .689 .563
.555 .582 .556 .573 .682 .709 .614
.637 .691 .708
.668 .802 .763
.634 .653 .692
.368 .500 .501
.466 .759 .861
.827 .832 .952
.616 .722 .741
.653 .636 .736
.588 .706 .751
.629 .693 .738
.749
.814
.747
.558
.842
.957
.740
.755
.770
.771
.491 .508 .515 .504 .614 .745 .484
.461 .469 .512 .510 .624 .737 .506
.558 .544 .549 .567 .609 .675 .461
.189 .200 .208 .237 .246 .281 .190
.612 .619 .675 .680 .616 .644 .668
.876 .881 .889 .902 .885 .875 .893
.528 .532 .529 .535 .597 .611 .514
.558 .563 .561 .570 .621 .642 .575
.547 .551 .567 .572 .609 .669 .532
.521 .528 .543 .555 .594 .634 .541
.701 .803 .747
.651 .704 .748
.627 .646 .712
.239 .433 .392
.749 .768 .819
.934 .939 .940
.614 .647 .776
.653 .620 .735
.673 .716 .764
.619 .674 .704
.752
.739
.734
.449
.886
.990
.791
.785
.791
.741
Table 9: Full uncertainty estimation results: Qwen3-14B, DeepSeek-Coder-V2, and Devstral-Small-2505
15
APPS Intro
Method
APPS Interview
HumanEval
MBPP
Average
AUROC PRAUC AUROC PRAUC AUROC PRAUC AUROC PRAUC AUROC PRAUC Qwen3-32B NL-derived baselines Mean Entropy Max Entropy Mean Probability Max Probability Consistency (BLEU) Consistency (VR) Symb. Clustering Three axes (ours) Top-5 Entropy (LEX) Pseudo Consistency (ALGO) Generated Tests (FUNC) Ensemble (ours) Top-5 + Tests + Pseudo Qwen3-Coder-30B NL-derived baselines Mean Entropy Max Entropy Mean Probability Max Probability Consistency (BLEU) Consistency (VR) Symb. Clustering Three axes (ours) Top-5 Entropy (LEX) Pseudo Consistency (ALGO) Generated Tests (FUNC) Ensemble (ours) Top-5 + Tests + Pseudo
.450 .460 .469 .467 .739 .799 .665
.561 .559 .602 .605 .797 .843 .776
.532 .491 .520 .528 .705 .762 .610
.291 .283 .316 .323 .479 .587 .477
.490 .523 .530 .549 .516 .574 .654
.868 .889 .900 .899 .885 .898 .922
.544 .532 .580 .573 .603 .663 .560
.617 .632 .658 .660 .664 .713 .660
.504 .502 .525 .529 .641 .700 .622
.584 .591 .619 .622 .706 .760 .709
.573 .671 .743
.644 .753 .804
.616 .642 .770
.369 .448 .571
.613 .621 .756
.902 .915 .938
.592 .611 .711
.657 .656 .724
.599 .636 .745
.643 .693 .759
.759
.884
.768
.603
.734
.944
.727
.738
.747
.792
.670 .780 .612 .633 .736 .689 .612
.710 .806 .659 .692 .781 .733 .697
.655 .722 .619 .643 .725 .667 .550
.458 .546 .518 .522 .544 .483 .434
.581 .678 .662 .667 .705 .578 .517
.957 .977 .951 .960 .981 .962 .960
.585 .570 .510 .536 .600 .528 .592
.627 .607 .610 .609 .619 .560 .616
.623 .688 .601 .620 .692 .616 .568
.688 .734 .684 .696 .731 .685 .677
.807 .721 .749
.822 .838 .809
.756 .610 .738
.580 .672 .576
.591 .671 .734
.967 .973 .973
.599 .602 .731
.629 .660 .722
.688 .651 .738
.750 .786 .770
.796
.852
.800
.614
.746
.980
.748
.767
.773
.803
Table 10: Additional full uncertainty estimation results: Qwen3-32B and Qwen3-Coder-30B
16