Semantic Voting: Execution-Grounded Consensus for LLM Code Generation
arXiv:2605.08680v1 [cs.SE] 9 May 2026
Shan Jiang∗ Zijian Yi∗ Chenguang Zhu The University of Texas at Austin [email protected]
Abstract LLM code-generation pipelines often sample multiple candidate programs and then select one final answer without access to a complete oracle. Existing pipelines mix textual voting, ranking, and execution-based agreement, and the relative contribution of each component remains unclear. We study 18 configurations across different models, thinking levels, and benchmarks, comparing output-pattern majority voting, weighted voting, MBR-Exec, and S EMANTIC VOTE—a method that clusters candidates by execution fingerprints on LLM-generated inputs. Three findings emerge. (1) The best execution-based selector exceeds output-pattern majority voting by 19–52 percentage points on every configuration, and every execution-based selector exceeds it by at least 18 points. (2) Once candidates are executed on diverse inputs, the aggregation rule has limited effect: S EMAN TIC VOTE, weighted voting, and MBR-Exec are statistically indistinguishable across all 18 configurations. Paired bootstrap tests find no significant difference between S EMANTIC VOTE and either alternative (p > 0.05 throughout); S EMANTIC VOTE differs from weighted voting by only −0.79 to +0.61 pp. The largest observed factor is input quality: sketch-based input generation, in which an LLM enumerates abstract input categories before instantiating concrete values, is consistently strongest in the input-strategy ablation, exceeding direct LLM input generation by 0.6–2.1 pp and random fuzzing or example-only inputs by up to 11.3 pp; this benefit transfers consistently to all three execution-based aggregators. (3) Thinking level interacts differently with the two selection families: on HumanEval+ Flash, deeper thinking improves majority voting by 12 pp, but execution-based methods stay flat or slightly degrade as candidate diversity falls. These results frame inference-time code selection as a signal-quality problem rather than an aggregation-rule problem: when oracles are unavailable, the behavioral evidence on which selection is based matters more than the rule used to aggregate it.
1
Introduction
Large language models (LLMs) are capable code generators, but selecting among multiple sampled candidate solutions remains difficult when no complete oracle is available. Existing systems combine textual voting, ranking heuristics, execution-based agreement, and learned reward models [Wang et al., 2023, Chen et al., 2023, Shi et al., 2022, Li et al., 2022], and the relative contribution of each component is unclear. We isolate two axes that prior work often conflates: (a) the value of richer behavioral evidence relative to all-or-nothing output-pattern voting, and (b) the relative importance of input quality and aggregation rule once selection uses behavioral evidence. A common baseline ported from chain-of-thought reasoning is output-pattern majority voting (MV) [Wang et al., 2023]: execute candidates on shared test inputs, discard any candidate that ∗ Equal contribution, ordered by last name.
Preprint.
crashes on any input, group the remaining candidates by exact equality of concatenated return-value strings, and pick the largest group. This all-or-nothing rule is sensitive to generated-input errors because generated inputs are imperfect behavioral probes. A candidate that is correct on the benchmark may still raise on a generated input outside the prompt’s intended domain, and majority voting then gives that candidate no vote. Conversely, if inputs miss a boundary case, an incorrect plurality can look identical to the correct behavior. The comparison therefore centers on how much behavioral evidence the selector preserves, rather than on execution alone. We use S EMANTIC VOTE to study this comparison. S EMANTIC VOTE retains exception-aware execution fingerprints instead of reducing selection to an all-success output-pattern key. Given N candidates, we execute each on D test inputs and record an execution fingerprint: the vector of outputs (or exception types) for each input. Programs with identical fingerprints form a semantic cluster, and the largest cluster’s representative is returned. S EMANTIC VOTE sits in a family of execution-based methods alongside weighted voting [Wang et al., 2025] and MBR-Exec [Shi et al., 2022]; we study the family rather than advocating for one member. A precondition for any execution-based method is a set of test inputs diverse enough to distinguish semantically different programs. We introduce sketch-based input generation: an LLM produces K abstract input sketches (e.g., “an empty list,” “a sorted list with duplicates,” “a list where all elements are negative”), each targeting a distinct behavioral equivalence class, and each sketch is instantiated M times with concrete values, yielding D = K × M inputs. Contributions. This is a findings paper. Across 18 configurations (3 Gemini models × 3 thinking levels × 2 benchmarks) we report three results: • Behavior over all-or-nothing voting. The best execution-based selector outperforms output-pattern majority voting by 19–52 pp on every configuration, and output-pattern MV consistently scores below Best-of-N . • Input quality over aggregation rule. Sketch-based input generation is consistently the strongest input strategy, exceeding direct LLM input generation by 0.6–2.1 pp and random fuzzing or example-only inputs by up to 11.3 pp. This gain transfers consistently to S EMANTIC VOTE, weighted voting, and MBR-Exec (at most 1.4 pp within strategy in the ablation). Once inputs are good, the three aggregators are statistically indistinguishable across all 18 configurations. Paired bootstrap tests find no significant difference between S EMANTIC VOTE and either alternative (p > 0.05 throughout); S EMANTIC VOTE differs from weighted voting by only −0.79 to +0.61 pp. • Different interactions with thinking depth. On HumanEval+ Flash, deeper thinking improves output-pattern MV (+12 pp), while execution-based methods stay flat or slightly decrease as candidate diversity falls. These results identify behavioral signal quality as the main factor in inference-time code selection. Scope of contribution. The paper is an empirical decomposition rather than a method paper. Bootstrap analysis shows S EMANTIC VOTE ≈ weighted voting ≈ MBR-Exec at p > 0.05 in all 18 configurations. The supported conclusions are that execution-based selection outperforms all-ornothing output-pattern voting, input quality drives most remaining variance, and thinking depth can affect the two selection families differently.
2
Background and motivation
Inference-time scaling for code. Sampling more candidates and applying better selection improves code generation accuracy beyond greedy decoding [Chen et al., 2021, Li et al., 2022, Snell et al., 2025]. This work studies candidate selection in settings where no complete oracle is available. Output-pattern majority voting. Self-consistency [Wang et al., 2023] samples multiple chainof-thought traces and picks the most common final answer. A common adaptation to code executes candidates on shared test inputs, discards candidates with any generated-input error, and clusters the survivors by exact equality of concatenated output strings. We treat this as a representative all-or-nothing output-pattern baseline: it uses execution, but it retains less behavioral evidence than exception-aware fingerprinting. 2
Algorithm 1 S EMANTIC VOTE Pipeline Require: Problem P , model M, budget N , sketches K, instantiations M Ensure: Selected solution s∗ 1: {s1 , . . . , sN } ← Sample(M, P, N ) // Generate candidates 2: S ← Filter({si }) // Remove syntactically invalid candidates 3: {x1 , . . . , xD } ← SketchInputs(M, P, K, M ) // Generate test inputs 4: for each si ∈ S do 5: fi ← [si (x1 ), . . . , si (xD )] // Compute fingerprint 6: end for 7: C ← ClusterByFingerprint({(si , fi )}) // Group by equality 8: C ∗ ← largest all-success cluster, else largest cluster // Prefer valid behavior 9: s∗ ← arg mins∈C ∗ |s| // Shortest program (Occam’s razor) 10: return s∗
Execution-based selection. CodeT [Chen et al., 2023] generates test cases alongside code and ranks programs by dual execution agreement. Shi et al. [2022] apply minimum Bayes risk with execution on provided examples. AlphaCode [Li et al., 2022] clusters candidates by outputs on example tests. Our work differs in two ways: (a) sketch-based LLM-generated inputs in place of provided examples, and (b) exact-equality clustering on full execution fingerprints rather than pairwise agreement or output on a few examples.
3
Method: S EMANTIC VOTE
3.1
Pipeline overview
S EMANTIC VOTE uses the LLM in exactly two places: generating candidate solutions (standard) and generating diverse test inputs via sketch-based input generation. The remaining steps—filtering, fingerprinting, clustering, and selection—are purely deterministic computation requiring no LLM calls. This separation is by design: it lets us isolate the effect of each component (input generation strategy, number of inputs, clustering method) in the ablation studies that follow, and it makes S EMANTIC VOTE a drop-in replacement for output-pattern voting at the same generation cost. Given a problem P and an LLM M, the pipeline proceeds in six steps (Algorithm 1). 3.2
Sketch-based input generation
We prompt the LLM to produce K input sketches: abstract descriptions of input categories targeting distinct behavioral equivalence classes. For a list-valued argument, example sketches include “an empty list,” “a sorted list with duplicates,” and “a list with all negative numbers.” Each sketch is instantiated M times with concrete values, yielding D = K × M inputs. The two-level structure (categories first, instances second) gives diversity across input types and redundancy within each type. 3.3
Execution fingerprinting
For each candidate si and input xj , we execute si (xj ) in a sandboxed subprocess with a 5second timeout and record one of: (ok, repr(result)) with floats rounded to 6 decimal places; (err, exception type) capturing the exception class name (not the message, which may be nondeterministic); or (timeout, TimeoutError). The fingerprint fi = [si (x1 ), . . . , si (xD )] is a tuple of such pairs. 3.4
Cluster selection
Two candidates are placed in the same cluster iff fi = fj (exact equality on all D inputs). We return the shortest program (Occam’s razor) from the largest cluster among those where all executions succeeded; if no all-success cluster exists, we fall back to the largest cluster overall. 3
Difference from output-pattern majority voting. Output-pattern MV, as adapted to code in prior work, also executes candidates, but it uses only all-success candidates and collapses each candidate to one concatenated output key. S EMANTIC VOTE instead keys on a tuple of (status, repr(value)) pairs, with floats rounded and container values serialized by Python after execution. This representation still distinguishes distinct values and exception types, but it does not discard a candidate’s entire behavioral trace after a single generated-input error. At selection time, S EMANTIC VOTE prefers allsuccess clusters when they exist; if none exist, it falls back to the largest exception-aware fingerprint cluster. Weighted voting and MBR-Exec use the same execution traces in different ways, which lets us compare aggregation rules while holding the behavioral signal fixed.
4
Experimental setup
Benchmarks. We use HumanEval+ (164 problems) and MBPP+ (378 problems), both from EvalPlus [Liu et al., 2023]. EvalPlus augments HumanEval with roughly 80× more tests and MBPP with roughly 35× more tests, making pass@1 evaluation more reliable than the original small test suites. MBPP+ is built from the hand-verified MBPP-sanitized subset rather than the full MBPP collection; the current EvalPlus release contains 378 tasks after removing broken or ill-formed tasks. We use the unmodified EvalPlus releases of both benchmarks. Models and thinking levels. We evaluate three preview Gemini endpoints, using the exact model IDs accepted by the API in our runs: gemini-3.1-pro-preview, gemini-3-flash-preview, and gemini-3.1-flash-lite-preview. Tables abbreviate these as 3.1 Pro, 3 Flash, and 3.1 Flash Lite. Each endpoint is evaluated at three thinking level settings (low, medium, high), which control the budget allocated to internal reasoning before output. The resulting 3 × 3 grid lets us vary model capability and inference-time thinking level independently, while staying within a single model family to control for architecture and training-data differences. Baselines.
We compare against the following baselines:
• Greedy: temperature 0, single sample. • Best-of-N : first syntactically valid candidate. • Majority voting: execute all candidates, discard any candidate with a generated-input error, group by exact concatenated output-pattern equality, return the shortest program from the largest group. • AST-normalized MV: parse each candidate into an AST, alpha-rename variables, strip docstrings, and group by canonical AST structure—a purely code-structural baseline that correctly groups candidates differing only in naming or formatting, but cannot distinguish semantically different programs with identical structure. • Weighted voting: output-pattern grouping with weights equal to per-candidate execution success rate. • MBR-Exec [Shi et al., 2022]: per-candidate score is the sum, over inputs, of how many other candidates agree on that input’s output or exception type; pick the highest-scoring. All non-greedy baselines share the same N cached candidates and the same test inputs as S EMAN TIC VOTE . Configuration. Unless stated otherwise, we sample N = 50 candidates at temperature 0.8 and remove syntactically invalid generations before selection. Sketch-based inputs use K = 10 and M = 5 (D = 50). Execution timeout is 5 s per (candidate, input). The input-strategy ablation in Section 5.1 uses cached N = 100 candidate pools held fixed across the four input strategies and three execution-based aggregators. Greedy is a per-model baseline. Greedy generations were cached per (model, prompt) without a thinking-level dimension before the cache key was extended to include thinking, so within a model the same single greedy sample populates the Greedy cells of every thinking-level row. The HumanEval+ Best-of-N column inherits the same cached generations and is also per-model on that benchmark; on MBPP+, the candidate pools were resampled later and the Best-of-N column does vary per thinking level. The three voting aggregators (MV, WV, MBR-Exec) and S EMANTIC VOTE are re-sampled per thinking level and reflect distinct candidate pools per cell. None of the paper’s claims compare Greedy 4
across thinking levels; Greedy and the HumanEval+ Best-of-N row are used only as per-model floors against which voting methods are measured.
5
Results
We organize the results around the two axes introduced above. Section 5.1 addresses input quality first, since it is the largest observed source of variance among execution-based methods. Sections 5.2–5.3 then compare aggregation rules at fixed input quality. Section 5.4 decomposes failures into generation and selection components.
5.1
Input generation strategy is the primary lever
A precondition for any execution-based method is a set of test inputs diverse enough to distinguish semantically different programs. We hold the candidate pool fixed and vary the input source across four strategies, evaluating each under all three execution-based selection methods: • Sketch (ours): K abstract categories, each instantiated M times (Section 3.2). • Direct LLM: D concrete inputs generated in one prompt, without the sketch abstraction. • Random: type-aware random fuzzing. • Example-only: only the example inputs from the problem description (typically 1–3).
Pro low (139)
Flash med (142)
Lite low (164)
Strategy
SV
WV
MBR
SV
WV
MBR
SV
WV
MBR
Sketch (ours) Direct LLM Random Example-only
99.3 97.8 94.2 95.0
100.0 98.6 94.2 95.0
98.6 97.8 95.0 94.2
97.2 95.8 89.4 86.6
97.9 95.8 89.4 86.6
96.5 94.4 88.7 85.9
95.1 94.5 93.9 90.9
95.1 94.5 93.9 90.9
94.5 93.9 93.9 90.9
Table 1: Pass@1 (%) on HumanEval+ across four input generation strategies and three executionbased aggregators, using cached N = 100 candidate pools held fixed within each configuration. Problem counts (in parentheses) restrict to problems where input generation succeeded for all strategies. Within each row, the three aggregators agree to within 1.4 pp; within a fixed model and aggregator, the strategy choice changes accuracy by up to 11.3 pp.
On Flash medium, sketch-based inputs reach 96.5–97.9% while direct LLM inputs reach 94.4–95.8%, a 1.4–2.1 pp sketch margin. Both LLM-generated input strategies outperform random fuzzing (88.7– 89.4%) and example-only inputs (85.9–86.6%) by roughly 6–11 pp. The main split is therefore not between sketching and direct concrete generation, but between LLM-generated diverse inputs and random fuzzing or the few examples in the prompt. The sketch abstraction still gives the strongest result in every column, suggesting that explicitly enumerating behavioral categories provides a small, consistent robustness gain over direct concrete generation. Within each strategy, the three execution-based aggregators agree to within 1.4 pp. The strategy×method interaction is therefore small relative to the input-source effect: the input source determines most of the behavioral-signal quality independently of how that signal is aggregated.
5.2
Aggregation rules: HumanEval+
We now fix the input source to sketch-based generation and compare aggregation rules across the full 3 × 3 grid (Table 2). 5
Model
Think
Greedy
Best-of-N
Maj. Vote
Wt. Vote
MBR-Exec
S EMANTIC VOTE
3.1 Pro
Low Medium High
93.9
97.0
75.0 77.4 76.2
98.8 98.2 97.6
97.6 96.3 95.7
98.8 97.6 97.0
3 Flash
Low Medium High
95.7
96.3
64.6 65.9 76.2
96.3 96.3 94.5
95.1 95.1 93.9
95.7 96.3 95.1
3.1 Flash Lite
Low Medium High
75.6
90.9
51.8 43.9 52.4
92.7 93.3 93.3
92.7 92.7 92.7
92.1 92.7 92.7
Table 2: Pass@1 (%) on HumanEval+ with N = 50 candidates. The three execution-based aggregators (WV, MBR-Exec, S EMANTIC VOTE) agree to within 1–2 pp on every configuration and are statistically indistinguishable (Appendix C); all three exceed output-pattern MV by 18–49 pp. Greedy and Best-of-N are reported as per-model baselines on this benchmark (one cached run per model reused across thinking levels; see Section 4).
Execution-based versus all-or-nothing voting. Output-pattern MV scores 43.9–77.4% across configurations, consistently below Best-of-N (90.9–97.0%); even greedy decoding on Pro (93.9%) exceeds MV on every Pro configuration. The gap to the best execution-based method ranges from 19 pp (Flash high vs. S EMANTIC VOTE) to 49 pp (Lite medium vs. WV) and is largest on weaker models, where more generated candidates hit at least one generated-input error. The main failure mode is the all-success filter: once every candidate has at least one error on the generated inputs, MV returns no candidate, whereas exception-aware execution methods can still use partial agreement. Effect of AST normalization. One alternative explanation is that exact output-pattern voting is too sensitive to representation and that a stronger non-execution normalization would close the gap. We test this with AST-normalized MV, which parses each candidate, alpha-renames variables, strips docstrings, and groups by canonical AST structure. On three representative configurations (Pro low, Flash medium, Lite low), AST-MV recovers most of the gap to output-pattern MV but still trails the best execution-based aggregator by 1–3 pp (Table 3). AST normalization rescues naming and formatting differences but cannot unify structurally different implementations that compute the same behavior, so its residual deficit is consistent with the need for behavioral evidence. Method
Pro low
Flash medium
Lite low
Output-pattern MV AST-normalized MV Best exec.-based (∆ AST − best exec.)
75.0 95.7 98.8 −3.1
65.9 94.5 96.3 −1.8
51.8 90.9 92.7 −1.8
Table 3: Pass@1 (%) on HumanEval+ for output-pattern MV, AST-normalized MV, and the best execution-based aggregator (WV or S EMANTIC VOTE). AST normalization closes 20–40 pp of the gap to execution-based methods but leaves a residual 2–3 pp deficit because it cannot unify structurally different but behaviorally equivalent programs.
Aggregation rules within the execution-based family. The three execution-based aggregators are within 1–2 pp on every configuration. Bootstrap analysis (10,000 problem-level resamples; Appendix C) shows the SV–WV difference is in [−0.79, +0.61] pp and not significant on any of the 18 configurations (p > 0.05); the SV–MBR-Exec difference is in [−0.61, +1.22] pp with the same conclusion. The convergence is consistent with the oracle-gap analysis (Section 5.4): with selection-failure rates already at 1.5–2.7%, the residual room for any aggregator to differentiate is small. Thinking level. On HumanEval+, output-pattern MV benefits from deeper thinking in the clearest Flash case (64.6% → 76.2%) and peaks at medium on Pro (75.0% → 77.4%). Execution-based methods stay flat or decrease slightly: on Pro, both WV and S EMANTIC VOTE drop from 98.8% (low) to 97.6%/97.0% (high); on Flash, WV drops from 96.3% (low) to 94.5% (high). Deeper thinking 6
Model
Think
Greedy
Best-of-N
Maj. Vote
Wt. Vote
MBR-Exec
S EMANTIC VOTE
3.1 Pro
Low Medium High
96.6
96.8 95.8 95.2
77.2 75.4 74.6
96.8 95.5 95.2
96.8 95.5 95.0
97.4 95.8 95.5
3 Flash
Low Medium High
95.2
95.5 92.6 92.9
59.5 51.3 52.9
95.0 92.3 91.5
95.0 92.3 91.3
95.2 91.8 91.0
3.1 Flash Lite
Low Medium High
91.3
91.0 91.8 91.5
39.4 41.0 42.6
91.5 92.1 92.1
91.3 91.8 91.3
90.7 91.3 91.5
Table 4: Pass@1 (%) on MBPP+ (375–378 problems per config, N = 50). The three execution-based aggregators are within 1–2 pp on every configuration and statistically indistinguishable (Appendix C); output-pattern MV trails by 19–52 pp. Greedy is reported as a per-model baseline (one cached run per model reused across thinking levels; Section 4). On MBPP+ the Best-of-N pool is resampled per thinking level, so the Best-of-N column does vary. Pass@1 (%)
Failure rate (%)
Config
Oracle
SV
WV
Gen.
SV sel.
WV sel.
Pro low Flash med Lite low
98.8 98.8 93.9
98.8 96.3 92.1
98.8 96.3 92.7
1.2 1.2 6.1
0.0 2.4 1.8
0.0 2.4 1.2
Avg. across 9 configs (HumanEval+) Avg. across 9 configs (MBPP+)
2.8 3.9
1.8 2.7
1.5 2.5
Table 5: Oracle-gap decomposition. Generation failures dominate; selection failures average 1.5–2.7% across all configurations. On Pro low, both SV and WV match the oracle ceiling.
appears to reduce candidate diversity, leaving fewer distinct behavioral clusters to exploit. This is an interaction effect rather than a universal monotonic gain, as the MBPP+ results below show. 5.3
Aggregation rules: MBPP+
The HumanEval+ patterns replicate: output-pattern MV scores 39.4–77.2%, the three executionbased aggregators stay within 1–2 pp of each other, and the gap is largest on Flash Lite (39–43% vs. 91–92%). On MBPP+, SV is slightly ahead on Pro (+0.27 to +0.53 pp) and WV is slightly ahead on Flash Lite (−0.53 to −0.79 pp), but bootstrap analysis finds no significant difference on any configuration. A candidate-pool effect is also visible: on Flash medium and high, greedy decoding (95.2%) exceeds the execution-based selection methods (91.0–92.3%), and the same pattern appears more mildly on Pro medium/high. Temperature-0.8 sampling introduces diversity at the cost of per-candidate quality, and selection cannot recover quality absent from the pool. 5.4
Oracle-gap decomposition
We decompose failures into generation failures (no correct candidate in the pool) and selection failures (a correct candidate exists but is not chosen). Selection failures average 1.5–2.7% across the 18 configurations; generation failures account for the rest. Generation is therefore the larger bottleneck, which helps explain the convergence among execution-based aggregators: with ≤3% of problems available for an aggregator-specific improvement, observable differences among aggregation rules are limited. Cluster diagnostics. S EMANTIC VOTE forms 1.2–1.5 clusters per problem on average, with the largest cluster containing 38–48 of the 50 candidates. Across the 9 HumanEval+ configurations, execution-based methods solve 31–81 more problems than output-pattern MV, and MV solves zero problems that all three execution-based methods miss. 7
D
5
10
20
30
50
Pass@1 (%) Avg. clusters/problem Avg. largest cluster
95.1 1.20 48.5
96.3 1.28 47.8
96.3 1.34 47.6
97.0 1.38 47.4
97.0 1.43 47.2
Table 6: D-scaling on HumanEval+ (Flash medium, N =50, sketch inputs). Pass@1 saturates at D≥30; cluster diagnostics show inputs continue to fragment behavior past saturation, but the marginal new clusters do not contain new correct programs.
Selection failures are systematic. Across the 18 configurations, S EMANTIC VOTE makes 119 selection errors on 52 unique problems; 34 problems recur in ≥ 2 configurations. Failed problems have an average largest-cluster size of 25.6 (versus 38–48 in the typical case), and greedy decoding rescues 88 of the 119 errors. The correct solution exists but is not the plurality behavior in the sampled pool—a limitation common to all consensus-based selection. Oracle computation in ablations. In Table 1, the oracle is computed on the subset of problems where input generation succeeded for all four strategies. Because this subset excludes problems where sketch inputs failed, accuracy on the full benchmark can occasionally exceed the subset oracle—an artifact of the restricted denominator, not a logical inconsistency. 5.5
D-scaling
In a separate Flash-medium sweep with N =50 and sketch inputs, we vary D ∈ {5, 10, 20, 30, 50} on HumanEval+ (Table 6). Pass@1 rises from 95.1% at D=5 to 96.3% at D=10 and saturates at 97.0% from D=30 onward. Even five well-chosen inputs already exceed output-pattern MV by 29 pp on Flash medium (95.1% vs. 65.9%). Beyond D=20, the marginal gain is negligible: the average number of behavioral clusters per problem grows only from 1.20 (D=5) to 1.43 (D=50), confirming that on this benchmark a small number of well-chosen inputs is enough to separate the dominant correct cluster from buggy variants. We use D=50 as a conservative default since execution is cheap relative to candidate generation. 5.6
Qualitative observations
Two patterns recur when S EMANTIC VOTE differs from output-pattern MV. Partial behavioral evidence is preserved: if every candidate raises on at least one generated input, output-pattern MV has no all-success survivor, while S EMANTIC VOTE can still select the plurality exception-aware fingerprint. Subtle bugs are separated: two divisors(n) candidates differing only by range(1, n+1) vs. range(1, n) agree on n ∈ {1, 2, 3}, but the sketch-generated input n = 6 separates the off-by-one variant.
6
Discussion
Contribution scope. Fingerprint clustering, weighted voting, and MBR-Exec all predate this work. Our contribution is the decomposition: input quality and aggregation rule are separate factors, and on these benchmarks input quality has the larger effect. The same input-quality gain transfers across SV, WV, and MBR-Exec, while no aggregator compensates for low-quality inputs. The convergence among SV/WV/MBR is itself part of the result: with selection-failure rates at 1.5–2.7%, room for any aggregator to differentiate is narrow, and this convergence is conditional on a shared, discriminative execution trace. Interpreting direct input generation. Direct LLM input generation is close to sketch generation in our runs, indicating that a capable model can often produce useful behavioral probes without an explicit sketch stage. The remaining sketch margin is consistent across all nine cells of Table 1. The narrower conclusion: sketching is a reliable way to make behavioral coverage explicit, but the larger effect is the move from sparse examples or random values to LLM-generated diverse inputs. 8
Practical guidance and overhead. The ablations suggest a simple ordering: replace prompt examples with generated probes first; prefer structured generation when the input domain has clear equivalence classes; increase D only after checking that new inputs create new useful clusters (most of the gain appears by D=10). Fingerprinting executes N ×D=2,500 sandboxed programs per problem in 10–30 s of local time versus 90–700 s of LLM time; sketch-based input generation costs one additional LLM call.
7
Related work
Self-consistency and majority voting. Wang et al. [2023] introduced self-consistency for chain-ofthought reasoning; a common code adaptation clusters candidates by exact output-pattern equality after execution [Chen et al., 2021]. Wang et al. [2025] aggregate ranked answer lists, but remain answer-text based and target reasoning rather than code. Execution-based code selection. CodeT [Chen et al., 2023] jointly generates tests and code; MBR-Exec [Shi et al., 2022] applies minimum Bayes risk on provided examples; Jiang et al. [2026a] demonstrate LLM effectiveness in code understanding and transformation tasks, using LLMs to guide deterministic code transformations; AlphaCode [Li et al., 2022] clusters outputs on example tests and scales to 106 candidates. We include MBR-Exec and compare against example-only inputs (Table 1). S EMANTIC VOTE differs by using sketch-based LLM-generated inputs and exact-equality fingerprint clustering. Test input generation. Fuzzing [Miller et al., 1990] and property-based testing [Claessen and Hughes, 2000] generate inputs automatically. LLM-based test generation has been studied by Chen et al. [2023], Hong et al. [2025] and Jiang et al. [2024]. Our sketch-based approach structures generation around behavioral categories rather than direct value generation. Program equivalence and inference-time scaling. Differential testing [McKeeman, 1998], program sketching [Solar-Lezama, 2013], and inference-time scaling for code [Snell et al., 2025] are related reference points; OBsmith [Jiang et al., 2026b] and APRIL [Zhong et al., 2025a,b] apply LLM-driven sketching and prompt optimization in adjacent settings. S EMANTIC VOTE improves the selection step within a fixed sampling budget.
8
Conclusion
Across three Gemini models, three thinking levels, and two benchmarks, inference-time code selection behaves primarily as a signal-quality problem. The best execution-based selector outperforms outputpattern majority voting by 19–52 pp on every configuration, and output-pattern MV consistently underperforms Best-of-N . Sketch-based input generation is consistently strongest in the inputstrategy ablation, with a 0.6–2.1 pp margin over direct LLM input generation and up to 11.3 pp over random fuzzing or example-only inputs. These input-source effects transfer across S EMANTIC VOTE, weighted voting, and MBR-Exec, which are statistically indistinguishable (p > 0.05) once inputs are fixed. Operationally, oracle-free selection benefits from exception-aware behavioral traces, input quality matters more than the execution-based aggregator, low-to-moderate thinking levels often work best for execution-based selection, and D ≈ 10 well-chosen inputs captures most of the gain in our sweep. S EMANTIC VOTE is the instrument used to evaluate these claims, but the main contribution is the decomposition. Limitations. We evaluate Gemini models on two Python benchmarks; other model families or languages require new runs and, for non-Python tasks, a new sandbox. Sketch-based input generation depends on the LLM’s domain familiarity, so specialized APIs may reduce the input-strategy advantage. Programs with intentional nondeterminism produce inconsistent fingerprints; we do not evaluate sandbox seeding as a mitigation. 9
References Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. CodeT: Code generation with generated tests. In International Conference on Learning Representations, 2023. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. Koen Claessen and John Hughes. QuickCheck: A lightweight tool for random testing of Haskell programs. In International Conference on Functional Programming, pages 268–279, 2000. Yang Hong, Shan Jiang, Yulei Fu, and Sarfraz Khurshid. On the effectiveness of large language models in writing Alloy formulas. arXiv preprint arXiv:2502.15441, 2025. Shan Jiang, Chenguang Zhu, and Sarfraz Khurshid. Generating executable oracles to check conformance of client code to requirements of JDK Javadocs using LLMs. arXiv preprint arXiv:2411.01789, 2024. Shan Jiang, Pranoy Kovuri, David Tao, and Zhixun Tan. CASCADE: LLM-powered JavaScript deobfuscator at Google. In ICSE-SEIP, 2026a. Shan Jiang, Chenguang Zhu, and Sarfraz Khurshid. OBsmith: LLM-powered JavaScript obfuscator testing. In ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA), 2026b. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with AlphaCode. Science, 378(6624):1092–1097, 2022. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by ChatGPT really correct? Rigorous evaluation of large language models for code generation. In Advances in Neural Information Processing Systems, 2023. William M McKeeman. Differential testing for software. Digital Technical Journal, 10(1):100–107, 1998. Barton P Miller, Louis Fredriksen, and Bryan So. An empirical study of the reliability of UNIX utilities. Communications of the ACM, 33(12):32–44, 1990. Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zettlemoyer, and Sida I Wang. Natural language to code translation with execution. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3533–3546, 2022. Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. In International Conference on Learning Representations, 2025. Armando Solar-Lezama. Program sketching. International Journal on Software Tools for Technology Transfer, 15:475–495, 2013. Weiqin Wang, Yile Wang, and Hui Huang. Ranked voting based self-consistency of large language models. In Findings of the Association for Computational Linguistics: ACL 2025, 2025. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations, 2023. Hua Zhong, Shan Jiang, and Sarfraz Khurshid. An approach for API synthesis using large language models. arXiv preprint arXiv:2502.15246, 2025a. Hua Zhong, Shan Jiang, and Sarfraz Khurshid. APRIL: API synthesis with automatic prompt optimization and reinforcement learning. arXiv preprint arXiv:2509.25196, 2025b.
10
A
Prompts
We provide the exact prompts used in our pipeline. All prompts are zero-shot with no in-context examples beyond the format specification. A.1
Candidate generation prompt
Used to generate N candidate solutions for each problem. The {prompt} placeholder is replaced with the benchmark problem’s function signature and docstring. Complete the following Python function. Write ONLY the function body (the lines that go inside the function). Do NOT repeat the function signature, docstring, or imports. Do NOT use markdown fences. {prompt} A.2
Sketch input generation prompt
Used to generate K diverse input sketches (abstract input categories) for a given function. The {K} placeholder is replaced with the desired number of sketches. You are generating diverse test inputs for a Python function. Function signature and description: {problem_description} Generate {K} diverse INPUT SKETCHES. Each sketch should target a fundamentally different equivalence class of behavior. Think about: - Edge cases (empty, single element, None, zero, negative) - Boundary values (max int, very long strings, deeply nested) - Typical cases (medium-sized, mixed types) - Special structure (sorted, reversed, all duplicates, alternating) For each sketch, provide: 1. A short description of what case it tests 2. A concrete Python expression that produces a valid input Output as JSON array: [{"description": "...", "input_expr": "..."}] Generate EXACTLY {K} sketches. A.3
Input variation prompt
Used to instantiate M − 1 additional concrete inputs from each sketch. Given this input sketch for a Python function: Description: {description} Example: {input_expr} 11
Function info: {problem_description} Generate {M} more concrete inputs that follow the SAME pattern but with different specific values. Output as a JSON array of Python expressions. Return ONLY the JSON array, no other text.
B
Detailed results
Across all 9 configurations on HumanEval+, execution-based methods (WV, MBR-Exec, S EMAN TIC VOTE) solve 31–81 more problems than output-pattern majority voting, while MV never solves a problem that any execution-based method misses. The oracle upper bound (percentage of problems where at least one candidate is correct) ranges from ∼93.9% (Flash Lite) to ∼99% (Pro), confirming that the remaining errors for the best methods are generation failures, not selection failures. Results on MBPP+ (Table 4) replicate the HumanEval+ patterns on a larger benchmark; SV and WV trade small leads across configurations and are never statistically distinguishable. The inputgeneration-strategy ablation (Table 1) and the D-scaling sweep (Section 5.5) together identify input quality as the critical component, with LLM-generated diverse inputs outperforming weaker input sources and input quality dominating both input quantity and the choice of aggregation rule. API usage. Each problem uses approximately 10K input tokens and 10K output tokens for N = 50 candidate-generation calls, plus approximately 500 input and 500 output tokens for sketch input generation. Dollar costs should be recomputed from the provider’s current pricing for the exact preview endpoint used; preview model availability and pricing can change. Execution and fingerprinting are local and do not require additional API calls.
C
Bootstrap confidence intervals
To rigorously assess whether S EMANTIC VOTE differs from the other execution-based aggregators in performance, we compute bootstrap confidence intervals using 10,000 problem-level resamples with replacement. For each resample, we compute pass@1 for the methods of interest and record the difference. Tables 7 and 8 report SV vs. WV on HumanEval+ and MBPP+ respectively; the SV vs. MBR-Exec comparison is summarized in the closing paragraph of this appendix.
Model
Think
SV
WV
∆ (SV−WV)
95% CI
p-value
3.1 Pro
Low Medium High
98.78 97.56 96.95
98.78 98.17 97.56
+0.00 −0.61 −0.61
[−0.00, +0.00] [−1.83, +0.00] [−1.83, +0.00]
1.000 0.370 0.376
3 Flash
Low Medium High
95.73 96.34 95.12
96.34 96.34 94.51
−0.61 +0.00 +0.61
[−3.05, +1.22] [−1.83, +1.83] [+0.00, +1.83]
0.394 0.655 0.361
3.1 Flash Lite
Low Medium High
92.07 92.68 92.68
92.68 93.29 93.29
−0.61 −0.61 −0.61
[−1.83, +0.00] [−1.83, +0.00] [−1.83, +0.00]
0.358 0.358 0.358
Table 7: Bootstrap analysis on HumanEval+ (164 problems, 10,000 resamples). No SV–WV difference is statistically significant at α = 0.05. All differences fall within ±0.61 percentage points.
12
Model
Think
SV
WV
∆ (SV−WV)
95% CI
p-value
3.1 Pro
Low Medium High
97.34 95.74 95.47
96.81 95.48 95.20
+0.53 +0.27 +0.27
[+0.00, +1.33] [−0.53, +1.06] [+0.00, +0.80]
0.133 0.388 0.368
3 Flash
Low Medium High
95.24 91.80 91.01
94.97 92.33 91.53
+0.26 −0.53 −0.53
[−0.53, +1.32] [−1.32, +0.00] [−1.32, +0.00]
0.394 0.137 0.135
3.1 Flash Lite
Low Medium High
90.74 91.27 91.53
91.53 92.06 92.06
−0.79 −0.79 −0.53
[−2.12, +0.26] [−1.85, +0.00] [−1.59, +0.53]
0.118 0.051 0.233
Table 8: Bootstrap analysis on MBPP+ (375–378 problems per config, 10,000 resamples). No SV–WV difference is statistically significant at α = 0.05, though Flash Lite medium (p = 0.051) is borderline. SV is consistently ahead of WV on Pro (+0.27 to +0.53 pp) but behind on Flash Lite (−0.53 to −0.79 pp).
Across all 18 configurations, the SV–WV difference never exceeds ±0.79 percentage points and is never statistically significant at α = 0.05. The same bootstrap procedure applied to SV vs. MBR-Exec yields differences in [−0.61, +1.22] pp, also non-significant on all 18 configurations (p > 0.05). This confirms that S EMANTIC VOTE, weighted voting, and MBR-Exec occupy the same statistical tier. By contrast, the gap between any execution-based method and output-pattern majority voting (18–52 points) is far larger, reinforcing our central finding that the choice of execution-based vs. output-pattern selection is more consequential than the choice among execution-based aggregation rules, as predicted by the signal-quality framing.
13