Test-Time Verification for Text-to-SQL via Outcome Reward Models Mattia Tritto1 , Giuseppe Farano1 , Dario Di Palma1 , Gaetano Rossiello2 , Fedelucio Narducci1 , Dharmashankar Subramanian2 , Tommaso Di Noia1 1
2
Polytechnic University of Bari, Bari, Italy IBM T.J. Watson Research Center, Yorktown Heights, NY, USA
Abstract
arXiv:2606.30851v1 [cs.CL] 29 Jun 2026
Improving the reliability of large language models (LLMs) at inference time is a central challenge in structured reasoning tasks such as Textto-SQL. Common test-time inference strategies, including Best-of-N sampling and Majority Voting, rely on heuristic signals such as execution success or output frequency, which provide limited semantic discrimination across candidate outputs. In this work, we study Outcome Reward Models (ORMs) as learned semantic scoring functions for test-time verification in Text-to-SQL. While ORMs have been previously explored for test-time scaling and alignment, their application to structured query generation remains underexplored. We introduce GradeSQL, a scalable framework for training task-specific ORMs via automated candidate generation and execution-based labeling, enabling verifier training without manual annotation. We integrate ORMs into a verification-driven Best-of-N pipeline and evaluate our approach on the BIRD and Spider benchmarks across multiple open-source LLM families. ORM-based selection consistently outperforms execution-based Best-of-N and Majority Voting, with gains of up to +4.33% on BIRD and +2.10% on Spider. We further show that ORMs scale effectively with larger candidate sets and yield stronger improvements on complex queries. Overall, our results demonstrate that ORM-based verification provides a simple, effective, and scalable alternative to heuristic test-time selection strategies for Textto-SQL. Code1 , datasets2 , and models3 are publicly available.
1
Introduction
As large language models (LLMs) are increasingly deployed in structured reasoning tasks, improving reliability at inference time has become a central 1
GradeSQL Framework GradeSQL Training Datasets 3 Pretrained GradeSQL ORMs 2
challenge. While scaling model size and training data has driven substantial progress, further gains increasingly depend on post-training techniques that leverage verification, feedback, and candidate selection mechanisms. In this setting, test-time inference plays a critical role: rather than producing a single output, models generate multiple candidates and rely on selection strategies to identify the most accurate solution. Text-to-SQL, the task of translating natural language questions into executable SQL queries, provides a natural testbed for studying test-time verification. It enables intuitive access to structured databases (Nascimento et al., 2024; Kim et al., 2020), but requires strict semantic correctness: small errors in query structure can lead to incorrect results. Despite strong progress with LLMs (Kumar, 2024; Pourreza and Rafiei, 2023), performance remains limited on complex queries involving multi-table joins, nested subqueries, and subtle constraints (Ma et al., 2024). This makes robust verification mechanisms essential. A common approach to improving performance is to increase test-time compute via strategies such as Best-of-N (BoN) sampling and Majority Voting (Cobbe et al., 2021; Sheng and Xu, 2025). These methods exploit generation diversity, but rely on heuristic signals such as execution success or output frequency, which provide only coarse proxies for semantic correctness. As candidate sets grow, these heuristics often saturate and fail to discriminate between semantically valid and invalid queries. Reward models offer a principled alternative by learning to score outputs according to task-specific correctness. In particular, Outcome Reward Models (ORMs) assign scalar scores to complete outputs and have been successfully used for test-time scaling and alignment in prior work (Cobbe et al., 2021). However, their application to structured reasoning tasks such as Text-to-SQL remains un-
derexplored, especially in terms of scalable data curation, task-specific training, and integration into practical inference pipelines. In this work, we investigate how ORMs can be effectively adapted and deployed as verifiers for Text-to-SQL. Our goal is not to introduce a new class of reward models, but to demonstrate how existing ORM paradigms can be trained and applied in a scalable way for structured query generation. A key challenge is data availability: training ORMs requires labeled candidate outputs, which are scarce for structured tasks. To address this, we introduce GradeSQL, a framework for scalable ORM training based on automated data synthesis. Given a natural language question and database schema, we generate diverse candidate SQL queries, label them via execution equivalence, and fine-tune a verifier to predict semantic correctness. At inference time, the trained ORM is used to re-rank candidates within a Bestof-N pipeline, replacing heuristic selection with learned scoring. We evaluate our approach on the BIRD and Spider benchmarks using multiple open-source LLM families. Results show that ORM-based verification consistently improves execution accuracy over heuristic baselines, with gains of up to +4.33% on BIRD and +2.10% on Spider. While absolute improvements are moderate, they are consistent across models and datasets, and are obtained without modifying the generator or requiring additional environment interaction at inference time. We further show that ORM-based selection scales more effectively with larger candidate budgets and yields stronger gains on complex queries. In summary, our contributions are: • ORM-based Verification for Text-to-SQL: We study Outcome Reward Models as learned semantic scorers for test-time candidate selection in structured query generation. • Scalable Data Curation Pipeline: We propose an automated framework for generating and labeling candidate SQL queries, enabling ORM training without manual annotation. • Empirical Analysis of Test-Time Verification: We provide a systematic evaluation of ORM-based selection across models, datasets, and inference regimes, showing consistent improvements over heuristic baselines.
2
Related Work
2.1
Text-to-SQL with LLMs
Text-to-SQL translates natural language questions into executable SQL queries, enabling natural language access to structured databases. The field has evolved from rule-based and logic-driven systems (Woods, 1972; Warren and Pereira, 1982) to neural semantic parsing approaches (Tang and Mooney, 2000; Xiao et al., 2016). The introduction of large-scale benchmarks such as Spider (Yu et al., 2018) enabled the development of schemaaware models based on pre-trained language models (PLMs) such as BERT and T5 (Devlin et al., 2019; Raffel et al., 2020), with architectures like RAT-SQL highlighting the importance of schema encoding (Wang et al., 2020). Recent advances with large language models (LLMs) have further improved performance through zero- and few-shot prompting, task decomposition, and retrieval-augmented generation (Kumar, 2024; Pourreza and Rafiei, 2023; Li et al., 2023). Despite these advances, Text-to-SQL remains challenging for complex queries involving multi-step reasoning and strict semantic correctness, motivating stronger verification mechanisms at inference time. 2.2
Test-Time Inference and Verification
Test-time inference (TTI) improves model outputs by allocating additional computation at inference without modifying model parameters (Snell et al., 2025). A common approach is to generate multiple candidate outputs and select among them. Best-ofN sampling and related strategies have been widely studied, including in mathematical reasoning where learned verifiers are used to select correct solutions from multiple candidates (Cobbe et al., 2021). Majority Voting further aggregates multiple outputs based on answer consistency. While these approaches are effective, many practical implementations rely on heuristic signals such as execution success, output agreement, or frequency, which may fail to capture semantic correctness in structured tasks. This has motivated the use of learned verification models that score candidate outputs based on task-specific criteria. 2.3
Reward Models and Learned Verifiers
Reward models provide a principled framework for scoring model outputs. Process Reward Models (PRMs) evaluate intermediate reasoning steps,
Stage 1: Candidates generation
Prompt: {question} {database_schema} Generate SQL.
LLM Generator
Stage 2: Data labeling
# Step by step solution: ## Select all the columns… ## From the tables… ## Final solution is: SELECT * FROM pilots
SELECT * FROM pilots
Generated CoTs
Final N SQL candidates
Stage 3: SFT
Candidate Queries
Label
SELECT * FROM pilots
Incorrect
SELECT ID FROM pilots
Correct
…
…
SELECT name FROM pilots Incorrect
Base LLM LoRA finetuning
Final ORM
Figure 1: Overview of the GradeSQL framework for training an ORM. The framework consists of three stages: (i) Candidate Generation, (ii) Data Labeling, and (iii) Supervised Fine-Tuning (SFT).
while Outcome Reward Models (ORMs) assign scalar scores to final outputs (Cobbe et al., 2021). ORMs have been successfully used for test-time scaling and alignment, particularly in reasoning tasks where multiple candidate solutions can be generated and ranked. More recent work explores generative verifiers (Zhang et al., 2025), where verification is framed as next-token prediction rather than explicit reward modeling, enabling scalable training without separate reward heads. These approaches further highlight the flexibility of learned verification mechanisms in post-training pipelines. In Text-to-SQL, learned verification remains relatively underexplored. Prior work has focused primarily on improving generation through prompt design, retrieval, or fine-tuning, rather than explicitly training verifiers. This leaves open the question of how to effectively train and deploy task-specific reward models for structured query verification. 2.4
ygoldi ∈ Ygold (Baek et al., 2025; Dai et al., 2025). Under a test-time inference setting, the goal extends from producing a single query to selecting the best query from a set of candidates generated by a large language model (LLM). Specifically, for a given question qi , an LLM produces a candidate set Ycandidate = {c1 , c2 , . . . , cN }, from which a verifier selects the most semantically faithful query. In this work, the verifier is an Outcome Reward Model (ORM), formalized as a scoring function ϕ(qi , cj ) ∈ [0, 1], which estimates the likelihood that candidate cj correctly captures the intent of qi . The final prediction is obtained via:
Data Curation for Verifier Training
A key challenge in training reward models is the availability of labeled data. Existing Text-to-SQL datasets are designed for generation rather than verification, typically providing a single gold query per input. Synthetic data generation approaches have been proposed to scale training data (Li et al., 2025a; Yang et al., 2024), but are not tailored to the needs of verifier training, which requires diverse candidate outputs and fine-grained correctness signals. In this work, we address this gap by introducing a scalable pipeline for generating and labeling candidate SQL queries, enabling the training of task-specific Outcome Reward Models for test-time verification.
3
Methodology
3.1
Problem Formulation
Given a natural language question qi ∈ Q and its associated database schema Σi , the Text-toSQL task aims to generate a correct SQL query
c∗ = arg max ϕ(qi , cj ). cj ∈Ycandidate
Training such a verifier poses two challenges: (i) constructing datasets with labeled candidate correctness, and (ii) assigning scores that reflect semantic alignment rather than surface heuristics. To address this, we leverage the LLM’s self-evaluation capabilities (Liu et al., 2024; Huang et al., 2024; Li et al., 2024), prompting it to assess candidate correctness and using the resulting logits as supervision signals. We implement this approach in the GradeSQL framework, illustrated in Figure 1. 3.2
GradeSQL Framework
Given a dataset D = {(qi , ygoldi )}ni=1 , GradeSQL trains an ORM through three stages. Stage 1: Candidate Generation For each question qi and schema Σi , a generator LLM G produces a set of N candidate SQL queries: Ycandidate = G(qi , Σi ) = {c1 , . . . , cN }.
3.3 ORM Inference and Probabilistic Scoring
Stage 1: Candidates generation Stage 2: Query selection Prompt: {question} {database_schema} Generate SQL.
At inference time, the ORM assigns a probabilistic score to each candidate SQL query. Given a prompt x = Prompt(Σ, q, c), the verifier defines:
Prompt: {question} {database_schema} {sql} Is the SQL correct?
LLM Generator Scoring candidates
Final ORM
Generated CoTs with Solution # Step by step solution: ## Select all the columns… ## From the tables… ## Final solution is: SELECT * FROM pilots Removing CoTs traces
Candidate Queries
Scores
SELECT ID FROM pilots
0.9
SELECT * FROM pilots
0.4
…
…
SELECT name FROM pilots
0.1
P (y | x; θ),
y ∈ {Yes, No},
and uses the probability of the “Yes” token as the candidate score:
SELECT * FROM pilots
Figure 2: ORM-based inference pipeline: candidate SQL queries are generated by an LLM and ranked by the ORM, with the highest-scoring query selected.
To promote diversity in the candidate set, generation is performed using stochastic decoding strategies, allowing the model to explore multiple plausible query formulations. This diversity is essential for effective verification, as it exposes the ORM to a broader range of correct and incorrect candidates, including semantically equivalent queries with different syntactic structures as well as nearmiss errors. Stage 2: Data Labeling Each candidate is executed on the database. Let R(cj ) denote the result set of query cj . A candidate is labeled as correct if R(cj ) = R(ygoldi ) and incorrect otherwise. Queries that raise execution errors are discarded. The labeling function is: if R(cj ) = R(ygoldi ), Yes ℓ(cj ) = No if R(cj ) ̸= R(ygoldi ), discarded if execution fails. Stage 3: Supervised Fine-Tuning The labeled dataset Dlabel = {(cj , ℓ(cj ))} is used to fine-tune a verifier LLM as a binary autoregressive classifier. Each input is constructed as x = Prompt(Σ, q, c), with target label l ∈ {Yes, No}. We fine-tune the model using LoRA (Hu et al., 2022) under the causal language modeling objective (Radford et al., 2019):
L(θ) = −
|s| X t=1
where s = [x; l].
Score(c) = P (yyes | x; θ).
SELECT ID FROM pilots
Final N SQL candidates
log Pθ (st | s<t ),
Rather than making a hard classification decision, this formulation yields a continuous confidence score that enables robust ranking of candidates (Figure 2). Probabilistic scoring offers fine-grained calibration across syntactically valid queries and avoids the brittleness of binary thresholding, making it well-suited for test-time verification. Interpretation as a Learned Verifier The ORM can be interpreted as a learned verifier that approximates semantic correctness beyond simple execution-based heuristics. While execution signals provide a binary notion of correctness, the ORM learns a softer scoring function that captures patterns across candidate queries, such as structural consistency, alignment with schema elements, and robustness to spurious matches. In particular, by training on diverse candidate sets that include both correct and incorrect queries, the ORM is exposed to fine-grained distinctions between semantically valid and invalid formulations. This allows it to assign higher confidence to candidates that are more likely to generalize beyond execution equivalence alone, especially in cases where multiple queries execute successfully but differ in semantic fidelity. As a result, the ORM complements heuristic selection by providing a more discriminative ranking signal over the candidate pool.
4
Experimental Setup
Datasets We evaluate on two cross-domain Textto-SQL benchmarks: Spider (Yu et al., 2018) and BIRD (Li et al., 2023). Both use disjoint databases across splits. For Spider, we train ORMs on the training set and evaluate on dev and test. For BIRD, we train on the training set and evaluate on the dev set due to the hidden test split. Metrics We use Execution Accuracy (EX), which measures whether predicted and gold queries return identical results, and Pass@N , which measures
Table 1: Reproducibility results of OmniSQL-7B. Maj = Majority Voting (temperature T =0.8, N =8); Gre = Greedy decoding (N =1). The table compares our reproduction (Our) with the original results (Origin) on the BIRD dev, Spider dev, and Spider test benchmarks.
Our Origin
BIRD dev
Spider dev
Spider test
Maj
Gre
Maj
Gre
Maj
Gre
66.95 66.10
64.41 63.90
83.95 81.60
82.11 81.20
85.61 88.90
84.49 87.90
whether at least one of the N generated candidates is correct. Baselines We compare against two test-time inference strategies: Majority Voting and executionbased Best-of-N , both selecting a single query from a candidate pool without using gold references. Majority Voting Candidates are executed and grouped by identical result sets. The largest group is selected and a query is sampled uniformly from it (Sheng and Xu, 2025). Execution-based Best-of-N Candidates are ranked using a simple execution heuristic that favors queries that execute successfully and return non-empty results (Chen et al., 2023). ORMs and Evaluation Setup We use OmniSQL7B (Li et al., 2025a) as the generator. For each question, the same set of N candidates is produced and shared across all methods. We compare Majority Voting, execution-based Best-of-N , and ORMbased Best-of-N over this fixed pool. This setup isolates the effect of the selection mechanism from generation quality. All methods operate on identical candidates and are evaluated under the same metric. ORM-based selection requires an additional offline training phase, so results reflect differences in selection quality rather than fully compute-matched end-to-end systems. To validate the generator, we reproduce OmniSQL results on Spider and BIRD. Table 1 shows deviations within 1% on BIRD dev and within 3% on Spider dev. Compute Considerations Heuristic baselines require no additional training, whereas ORM-based selection incurs offline verifier training. At inference time, all methods operate on the same candidate pool. The trade-off is between upfront training cost and improved semantic discrimination during selection. In our setting, training is amortized across queries and requires no additional environment interaction beyond candidate generation and
Table 2: Execution accuracy (%) of OmniSQL-7B on BIRD dev, Spider dev, and Spider test. ∆ indicates the gain over the baseline (N =1). Best and runner-up results are in bold and underlined, respectively. Values marked with † are not McNemar significant (p < 0.05), and those with * lose significance after Bonferroni correction. Method
BIRD dev EX
Baseline (N=1) Majority Voting (N=32) Best-of-N (N=32) Execution-based ORM-based
∆
63.89 – 66.95 +3.06
Spider dev EX 82.40 83.75∗
∆
Spider test EX
∆
– 84.02 – +1.35 85.47 +1.45
66.04 +2.15 82.79†∗ +0.39 85.14 +1.12 68.90 +5.01 84.53∗ +2.13 87.47 +3.45
execution. Reproducibility Details Experiments were run on nodes with Intel Xeon 8358 CPUs, 512 GB RAM, and 4 NVIDIA A100 GPUs (64 GB). Candidates were generated using vLLM, and ORM inference used Hugging Face transformers. All runs use seed 42.
5
Results and Discussion
We evaluate our framework across seven opensource LLMs: Qwen2.5-1.5B/7B-Instruct, Granite3.3-2B/8B-Instruct, Llama-3.2-1B, Llama-3.1-8BInstruct, and OmniSQL-7B. Experiments are conducted on BIRD dev (1,534 samples), Spider dev (1,034 samples), and Spider test (2,147 samples). ORM training data is constructed from the original training splits (9,428 queries for BIRD and 9,000 for Spider), yielding 9,411 and 8,960 queries after light preprocessing. For each question, OmniSQL-7B generates N = 32 SQL candidates, following prior work (Li et al., 2025b) as a trade-off between performance and computational cost (Wang et al., 2023; Lewkowycz et al., 2022). Invalid queries are discarded, and remaining candidates are labeled based on execution equivalence with the gold query, resulting in 82,640 labeled samples for BIRD and 50,073 for Spider. ORM vs. Test-Time Baselines Table 2 compares ORM-based Best-of-N with execution-based Bestof-N (ex-BoN) and Majority Voting (Maj), using OmniSQL-7B as the generator. Statistical significance is assessed using the McNemar test with Bonferroni correction (Dror et al., 2018). ORM-based selection achieves the highest execution accuracy across all datasets. Except for Spider dev (likely due to smaller sample size), improvements remain statistically significant. On BIRD dev, ORM reaches 68.90%, outper-
Table 3: Dataset statistics for BIRD and Spider (training splits). We report (i) the number of questions before and after de-duplication (with removals ∆), and (ii) the final size of the labeled sets generated by sampling N = 32 candidates per question. Percentages indicate the class distribution (Incorrect / Correct) in the imbalanced set; the balanced set is 50/50.
Table 5: Execution accuracy (%) of ORM models on BIRD dev, Spider dev, and Spider test. Each cell shows results on Unbalanced and Balanced sets with ∆ (green ↑ = gain, red ↓ = drop). Ordered by BIRD Unbalanced; best in bold, runner-up underlined. ORM Model Granite-3.3-2B-Instruct
BIRD (train) Questions # (pre) # (post) Removed ∆ (%) Labeled sets Imbalanced size Class split (% inc / cor) Balanced size (50/50)
Spider (train) Llama-3.2-1B-Instruct
9,428 9,411 17 (0.18%)
9,000 8,960 40 (0.44%)
Granite-3.3-8B-Instruct Llama-3.1-8B-Instruct
82,640 41.37 / 58.63 30,686
50,073 30.60 / 69.40 17,834
Table 4: ORM performance on BIRD and Spider with the unbalanced dataset, reporting Execution Accuracy (%) on dev and test sets. ORM Model
Qwen2.5-1.5B-Instruct
Bird dev
Spider dev
Spider test
Granite-3.3-2B-Instruct Granite-3.3-8B-Instruct
66.88 68.25
82.79 84.42
85.14 86.91
Llama-3.2-1B-Instruct Llama-3.1-8B-Instruct
67.67 68.32
84.04 82.30
86.31 85.89
Qwen2.5-1.5B-Instruct Qwen2.5-7B-Instruct
68.12 68.58
84.33 84.33
86.17 86.91
OmniSQL-7B
68.64
84.42
86.63
forming Maj (66.95%) and ex-BoN (66.04%), with a +5.01 gain over the N =1 baseline. On Spider dev, ORM achieves 84.53% (+2.13 over Maj), and on Spider test 87.47%, exceeding both Maj (85.47%) and ex-BoN (85.14%) by a wide margin. Dataset Imbalance The resulting datasets exhibit class imbalance: Spider contains 69.4% correct candidates, while BIRD is more balanced (58.63% correct). To study its impact, we construct balanced datasets by downsampling to a 50/50 split per question. Statistics are summarized in Table 3. Using these datasets, we address three research questions: (i) cross-family ORM trainability, (ii) the effect of dataset balancing, and (iii) ORM effectiveness relative to test-time baselines. Cross-Family Trainability Table 4 reports ORM performance trained on unbalanced data across all backbones. Results are tightly clustered: BIRD dev ranges from 66.88–68.64, Spider dev from 82.30–84.42, and Spider test from 85.14–86.91. Performance differences across model families and scales are modest, indicating that ORMs train reliably across heterogeneous architectures. Larger models yield only minor gains, suggesting ORM effectiveness is not strongly dependent on parameter count. Rankings are stable across datasets
Qwen2.5-7B-Instruct OmniSQL-7B
BIRD dev
Spider dev
Spider test
Unbalanced → Balanced
Unbalanced → Balanced
Unbalanced → Balanced
66.88 → 66.17
82.79 → 82.79
85.14 → 85.14
(-0.71 ↓)
(0.00)
(0.00)
67.67 → 67.21
84.04 → 83.56
86.31 → 86.54
(-0.46 ↓)
(-0.48 ↓)
(+0.23 ↑)
68.12 → 67.28
84.33 → 83.56
86.17 → 86.31
(-0.84 ↓)
(-0.77 ↓)
(+0.14 ↑)
68.25 → 68.38
84.42 → 83.85
86.91 → 87.00
(+0.13 ↑)
(-0.57 ↓)
(+0.09 ↑)
68.32 → 67.86
82.30 → 84.11
85.89 → 86.96
(-0.46 ↓)
(+1.81 ↑)
(+1.07 ↑)
68.58 → 68.19
84.33 → 84.14
86.91 → 86.68
(-0.39 ↓)
(-0.19 ↓)
(-0.23 ↓)
68.64 → 68.90
84.42 → 84.53
86.63 → 87.47
(+0.26 ↑)
(+0.11 ↑)
(+0.84 ↑)
and splits, demonstrating robustness and familyagnostic generalization. Effect of Dataset Balancing Table 5 compares ORMs trained on unbalanced versus balanced data. Balancing produces mixed effects on dev sets but yields more consistent improvements on Spider test, particularly for Llama-3.1-8B (+1.07) and OmniSQL-7B (+0.84). Overall changes remain small (within ±1.2), indicating ORM stability under skewed label distributions. Importantly, balanced datasets substantially reduce training size (from 80k→30k for BIRD and 50k→18k for Spider), lowering training cost without degrading performance. In several cases, balanced training even improves accuracy. We therefore adopt the balanced configuration for subsequent comparisons. Summary ORMs (i) train reliably across model families, (ii) remain stable under label imbalance while benefiting from balanced data, and (iii) consistently outperform widely used test-time heuristics. These results establish ORM-based selection as a stronger and more principled alternative to Majority Voting and execution-based Best-of-N for test-time inference in Text-to-SQL.
6
Ablation Studies
We conduct ablations to better understand the behavior of Outcome Reward Models (ORMs) under different design and scaling choices. In particular, we study: (i) the effect of candidate pool size N , (ii) ORM performance with larger generators, (iii) sensitivity to training prompts, (iv) scaling ORMs beyond 7B parameters, and (v) the impact of finetuning objectives.
Table 6: Execution accuracy (%) and Pass@N of test-time strategies (execution-based BoN, Majority Voting, ORM-based BoN) across varying N values on the BIRD dev, Spider dev, and Spider test benchmarks. Columns are color-coded from light yellow (lowest) to dark orange (highest) performance. Execution-based BoN
N 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1
Majority Voting
ORM-based BoN
Pass@N
BIRD dev
Spider dev
Spider test
BIRD dev
Spider dev
Spider test
BIRD dev
Spider dev
Spider test
BIRD dev
Spider dev
Spider test
66.04 66.04 66.04 65.97 65.97 65.97 65.97 65.97 65.97 65.97 65.97 65.91 65.91 65.91 65.91 65.91 65.91 65.84 65.84 65.84 65.84 65.84 65.71 65.71 65.71 65.58 65.45 65.38 65.32 65.12 64.86 63.89
82.79 82.79 82.88 82.88 82.88 82.88 82.88 82.88 82.88 82.88 82.88 82.88 82.88 82.88 82.88 82.88 82.88 82.88 82.88 82.88 82.88 82.88 82.88 82.88 82.88 82.79 82.79 82.79 82.88 82.88 82.79 82.40
85.14 85.14 85.14 85.14 85.14 85.14 85.14 85.14 85.14 85.14 85.14 85.14 85.14 85.14 85.14 85.14 85.14 85.14 85.10 85.10 85.05 85.05 85.05 85.05 85.05 85.05 85.00 85.00 85.00 84.91 84.63 84.02
66.95 67.08 67.01 66.95 67.01 67.14 66.75 66.75 67.01 67.08 66.88 66.75 67.01 67.08 67.01 67.08 66.95 67.14 67.01 67.14 66.88 66.82 66.69 66.49 66.95 66.62 66.43 65.97 65.32 65.19 64.41 63.89
83.75 83.75 83.66 83.56 83.75 83.85 83.75 83.85 83.95 84.14 83.95 83.85 83.75 83.85 83.76 83.95 83.66 83.85 83.85 83.66 83.85 83.66 84.14 84.04 83.95 84.04 83.66 83.95 83.66 83.56 83.08 82.40
85.47 85.47 85.42 85.42 85.47 85.47 85.33 85.51 85.56 85.47 85.42 85.47 85.56 85.37 85.65 85.61 85.61 85.42 85.61 85.56 85.51 85.19 85.56 85.65 85.61 85.70 85.75 85.65 85.47 85.33 84.44 84.02
68.90 68.58 68.58 68.51 68.58 68.58 68.64 68.71 68.77 68.71 68.90 68.84 68.77 68.77 68.84 68.77 68.77 68.71 68.64 68.64 68.71 68.64 68.64 68.51 68.51 68.38 67.86 67.47 67.34 66.75 66.04 63.89
84.53 84.53 84.62 84.62 84.62 84.62 84.62 84.62 84.62 84.62 84.62 84.62 84.62 84.62 84.62 84.72 84.53 84.53 84.53 84.53 84.44 84.33 84.33 84.24 84.33 84.33 84.24 84.14 84.04 84.04 83.66 82.40
87.47 87.47 87.47 87.42 87.38 87.38 87.38 87.38 87.33 87.33 87.33 87.28 87.33 87.33 87.38 87.33 87.33 87.28 87.19 87.19 86.96 86.87 86.91 86.91 86.82 86.77 86.54 86.49 86.49 86.17 85.75 84.02
80.57 80.25 80.25 80.12 80.05 79.92 79.92 79.79 79.53 79.47 79.33 79.14 79.07 78.88 78.88 78.68 78.49 78.23 77.90 77.84 77.71 77.25 76.99 76.60 76.14 75.36 74.58 73.79 72.82 70.86 68.84 63.89
91.68 91.68 91.68 91.68 91.68 91.59 91.59 91.59 91.59 91.49 91.39 91.20 91.01 90.91 90.81 90.81 90.72 90.52 90.52 90.43 90.43 90.23 90.04 89.85 89.65 89.26 89.17 88.59 88.30 87.43 86.07 82.40
93.29 93.20 93.15 93.11 93.01 93.01 92.87 92.87 92.78 92.78 92.73 92.73 92.69 92.59 92.55 92.50 92.40 92.27 92.13 92.08 91.94 91.71 91.66 91.62 91.48 91.06 90.82 90.54 90.13 89.10 87.66 84.02
Model
Baseline
Ex-BoN
Majority Voting
ORM-BoN
Pass@32
BIRD dev
OmniSQL-7B OmniSQL-14B OmniSQL-32B
63.89 64.28 64.02
66.04 66.75 66.10
66.95 67.01 63.95
68.90 69.04 69.30
80.57 80.90 82.20
Spider dev
OmniSQL-7B OmniSQL-14B OmniSQL-32B
82.40 82.01 83.17
82.79 82.20 83.37
83.75 83.56 83.08
84.33 84.62 84.43
91.68 92.75 92.84
Spider test
Table 7: Execution accuracy (%) and Pass@32 on the BIRD dev, Spider dev, and Spider test. Candidate queries are generated by OmniSQL models with 7B, 14B, and 32B parameters, while the ORM is implemented using OmniSQL-7B. Best scores per column are shown in bold.
OmniSQL-7B OmniSQL-14B OmniSQL-32B
84.02 84.91 84.30
85.14 85.28 84.72
85.47 84.86 86.07
87.33 87.42 87.19
93.29 93.53 94.08
Effect of Candidate Pool Size N Table 6 shows that ORM-based selection consistently outperforms execution-based Best-of-N and Majority Voting across all values of N on BIRD and Spider. While heuristic methods saturate early, ORM performance continues to improve as N increases and more closely approaches the oracle upper bound defined by Pass@N . Gains are largest on challenging queries (Figure 3), where semantic discrimination is most critical. These results indicate that ORMs more effectively leverage increased test-time compute.
Table 8: Ablation study on verification prompt design for ORM on BIRD dev. Execution accuracy (%) by query difficulty. Generator: OmniSQL-7B; ORM: Qwen2.5-7B-Instruct. Best scores in bold. Prompt Variant
Simple
Moderate
Challenging
Total
Instruction Data-Only SQL-Only Data + SQL
72.65 72.76 72.86 73.08
59.70 59.48 63.36 62.50
51.32 51.72 53.79 50.34
66.72 66.75 68.19 67.73
ORMs with Larger Generators Table 7 evaluates ORM-based selection with increasingly large OmniSQL generators (7B, 14B, 32B). Across BIRD and Spider, ORMs consistently outperform heuristic baselines and maintain stable improvements as generator size increases. While larger generators improve absolute performance, the relative gains from ORM-based selection persist, indicating that ORM benefits are complementary to generator scaling. Impact of Training Prompt Table 8 studies prompt design for ORM training. Differences are negligible for simple queries but become pronounced for moderate and challenging ones. The SQL-only prompt consistently yields the best over-
62
73.5
61
73.0
60
72.5
59
72.0
58
71.5
Strategies
ex-BoN Maj ORM-BoN
71.0 70.5
Table 10: Ablation study comparing autoregressive and Binary Cross-Entropy (BCE) loss fine-tuning on BIRD dev across query difficulty levels. Both the generator and ORM model are OmniSQL-7B. Reported metric is Execution Accuracy (%). Best results are in bold.
Execution Accuracy (%)
63
74.0
Execution Accuracy (%)
74.5
0
4
57
Strategies
ex-BoN Maj ORM-BoN
56
8 12 16 20 24 28 32
0
Number of Generations
(a) Simple
4
8 12 16 20 24 28 32
Number of Generations
(b) Moderate 69
Execution Accuracy (%)
Execution Accuracy (%)
68
Moderate
Challenging
Total
Majority Voting BCE FT Autoregressive FT
72,86 72,86 74.15
59,70 59,91 62.93
52,41 51,72 54.48
66,95 66,95 68.90
67
52 51
66
50 Strategies
49 48
Simple
captures semantic correctness than direct classification.
54 53
Method
ex-BoN Maj ORM-BoN
0
4
65
Strategies
ex-BoN Maj ORM-BoN
64
8 12 16 20 24 28 32
0
Number of Generations
(c) Challenging
4
8 12 16 20 24 28 32
Number of Generations
(d) Overall
Figure 3: Execution accuracy on BIRD dev as a function of N , comparing execution-based Best-of-N , Majority Voting, and ORM-based Best-of-N , stratified by query difficulty. Table 9: Impact of ORM model size with the generator fixed to OmniSQL-7B. Execution Accuracy (EX, %) and Pass@32 are reported on BIRD dev, Spider dev, and Spider test. Best results per metric and dataset are shown in bold. ORM OmniSQL-7B OmniSQL-14B OmniSQL-32B
BIRD dev
Spider dev
Spider test
EX
Pass@32
EX
Pass@32
EX
Pass@32
68.90 69.23 68.19
80.57 80.57 80.57
84.53 85.01 84.23
91.68 91.68 91.68
87.47 87.84 86.91
93.29 93.29 93.29
all accuracy, outperforming instruction-style and data-only variants. This suggests that exposing the verifier directly to SQL structure provides the most effective signal for semantic verification. Scaling ORMs Beyond 7B Table 9 shows that increasing ORM size beyond 7B parameters yields only marginal and inconsistent improvements. While a 14B ORM occasionally improves over 7B, a 32B ORM often underperforms, indicating diminishing returns. Overall, lightweight ORMs achieve a favorable balance between performance and efficiency, and larger verifiers cannot compensate for generator limitations. Fine-Tuning Objective Table 10 compares autoregressive fine-tuning with Binary Cross-Entropy (BCE). Autoregressive training consistently outperforms BCE and Majority Voting, with the largest gains observed on moderate and challenging queries. These results indicate that modeling verification as an autoregressive generation task better
Summary Across all ablations, ORMs demonstrate stable behavior under varying candidate budgets, generator scales, prompts, and training objectives. Performance is primarily driven by the quality of feedback and verification signal rather than verifier size, reinforcing the effectiveness of lightweight, autoregressive ORMs for test-time verification.
7
Conclusion
In this work, we studied the use of Outcome Reward Models (ORMs) as learned verification mechanisms for test-time inference in Text-to-SQL. Rather than introducing new reward modeling techniques, our goal was to investigate how existing ORM paradigms can be effectively adapted to structured query generation, where semantic correctness is critical. We introduced GradeSQL, a scalable pipeline for training task-specific ORMs through automated candidate generation and executionbased labeling, enabling verifier training without manual annotation and supporting deployment in standard Best-of-N inference pipelines. Across the BIRD and Spider benchmarks, ORM-based selection consistently improves execution accuracy over heuristic baselines such as execution-based Best-of-N and Majority Voting. While absolute gains are moderate, they are stable across models and datasets, and become more pronounced in challenging queries and larger candidate regimes, where heuristic methods tend to saturate. Overall, our findings suggest that learned verification provides a simple and effective complement to heuristic test-time selection strategies in Text-to-SQL, highlighting the importance of data curation and task-specific training for scalable verifier-based inference in structured reasoning tasks. We release our code, datasets, and trained models to support reproducibility.
Limitations ORM-based selection requires an additional offline training phase, unlike heuristic baselines such as Majority Voting and execution-based Best-of-N . While this cost is amortized across queries, our comparison is not fully compute-matched end-toend. Moreover, the observed gains, although consistent, are moderate (2–5%) and mainly arise in challenging queries and larger candidate regimes, indicating improvements in selection rather than in the underlying generation capability. Our approach relies on execution equivalence as a proxy for semantic correctness, which may fail to distinguish semantically equivalent queries or may reward spurious matches. In addition, evaluation is limited to clean benchmark datasets (Spider and BIRD), and performance in real-world settings with noisy schemas or ambiguous queries remains unclear. Finally, we focus on binary Outcome Reward Models; richer verification signals, such as fine-grained or process-level feedback, are not explored.
Ethical Considerations This work focuses on improving the reliability of LLM-based systems for structured data access through Text-to-SQL, with potential benefits for accessibility and decision support. However, incorrect or misleading SQL generation may lead to faulty data retrieval and downstream decisions, particularly in high-stakes domains. While ORMbased verification improves selection quality, it relies on execution-based supervision, which may inherit biases or errors present in underlying data or schemas. Our approach does not introduce new personal data or sensitive information, but care should be taken when deploying such systems on proprietary or privacy-sensitive databases. Finally, increased use of test-time compute and model-based verification may have environmental and cost implications. We encourage future work on robust evaluation in real-world settings and on developing more transparent and accountable verification mechanisms.
References Jinheon Baek, Horst Samulowitz, Oktie Hassanzadeh, Dharmashankar Subramanian, Sola Shirai, Alfio Gliozzo, and Debarun Bhattacharjya. 2025. Knowledge base construction for knowledge-augmented text-to-sql. In ACL (Findings), pages 26569–26583. Association for Computational Linguistics.
Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2023. Codet: Code generation with generated tests. In ICLR. OpenReview.net. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. CoRR, abs/2110.14168. Yaxun Dai, Haiqin Yang, Hao Mou, and Pingfu Chao. 2025. PARSQL: enhancing text-to-sql through SQL parsing and reasoning. In ACL (Findings), pages 661–681. Association for Computational Linguistics. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT (1), pages 4171–4186. Association for Computational Linguistics. Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker’s guide to testing statistical significance in natural language processing. In ACL (1), pages 1383–1392. Association for Computational Linguistics. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In ICLR. OpenReview.net. Hui Huang, Yingqi Qu, Jing Liu, Muyun Yang, Bing Xu, Tiejun Zhao, and Wenpeng Lu. 2024. Self-evaluation of large language model based on glass-box features. In EMNLP (Findings), pages 5813–5820. Association for Computational Linguistics. Hyeonji Kim, Byeong-Hoon So, Wook-Shin Han, and Hongrae Lee. 2020. Natural language to SQL: where are we today? Proc. VLDB Endow., 13(10):1737– 1750. Pranjal Kumar. 2024. Large language models (llms): survey, technical frameworks, and future challenges. Artif. Intell. Rev., 57(9):260. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay V. Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. 2022. Solving quantitative reasoning problems with language models. In NeurIPS. Haoyang Li, Shang Wu, Xiaokang Zhang, Xinmei Huang, Jing Zhang, Fuxin Jiang, Shuai Wang, Tieying Zhang, Jianjun Chen, Rui Shi, Hong Chen, and Cuiping Li. 2025a. Omnisql: Synthesizing highquality text-to-sql data at scale. Proc. VLDB Endow., 18(11):4695–4709.
Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin Chen-Chuan Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2023. Can LLM already serve as A database interface? A big bench for largescale database grounded text-to-sqls. In NeurIPS.
parsing. In EMNLP, pages 133–141. Association for Computational Linguistics. Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2020. RAT-SQL: relation-aware schema encoding and linking for textto-sql parsers. In ACL, pages 7567–7578. Association for Computational Linguistics.
Moxin Li, Wenjie Wang, Fuli Feng, Fengbin Zhu, Qifan Wang, and Tat-Seng Chua. 2024. Think twice before trusting: Self-detection for large language models through comprehensive answer reflection. In EMNLP (Findings), pages 11858–11875. Association for Computational Linguistics.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In ICLR. OpenReview.net.
Yanyang Li, Michael R. Lyu, and Liwei Wang. 2025b. Learning to reason from feedback at test-time. In ACL (1), pages 5241–5253. Association for Computational Linguistics.
David H. D. Warren and Fernando C. N. Pereira. 1982. An efficient easily adaptable system for interpreting natural language queries. Am. J. Comput. Linguistics, 8(3-4):110–122.
Weize Liu, Guocong Li, Kai Zhang, Bang Du, Qiyuan Chen, Xuming Hu, Hongxia Xu, Jintai Chen, and Jian Wu. 2024. Mind’s mirror: Distilling self-evaluation capability and comprehensive thinking from large language models. In NAACL-HLT, pages 6748–6763. Association for Computational Linguistics.
William Woods. 1972. The lunar sciences natural language information system. BBN report.
Limin Ma, Ken Q. Pu, and Ying Zhu. 2024. Evaluating llms for text-to-sql generation with complex SQL workload. CoRR, abs/2407.19517. Eduardo Nascimento, Grettel M. García, Lucas Feijó, Wendy Victorio, Yenier Torres Izquierdo, Aiko R. de Oliveira, Gustavo M. C. Coelho, Melissa Lemos, Robinson Luiz Souza Garcia, Luiz André P. Paes Leme, and Marco A. Casanova. 2024. Text-to-sql meets the real-world. In ICEIS (1), pages 61–72. SCITEPRESS.
Chunyang Xiao, Marc Dymetman, and Claire Gardent. 2016. Sequence-based structured prediction for semantic parsing. In ACL (1). The Association for Computer Linguistics. Jiaxi Yang, Binyuan Hui, Min Yang, Jian Yang, Junyang Lin, and Chang Zhou. 2024. Synthesizing text-tosql data from weak and strong llms. In ACL (1), pages 7864–7875. Association for Computational Linguistics.
Mohammadreza Pourreza and Davood Rafiei. 2023. DIN-SQL: decomposed in-context learning of textto-sql with self-correction. In NeurIPS.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir R. Radev. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In EMNLP, pages 3911–3921. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog.
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. 2025. Generative verifiers: Reward modeling as next-token prediction. In ICLR. OpenReview.net.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67. Lei Sheng and Shuai-Shuai Xu. 2025. CSC-SQL: corrective self-consistency in text-to-sql via reinforcement learning. CoRR, abs/2505.13271. Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2025. Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. In ICLR. OpenReview.net. Lappoon R. Tang and Raymond J. Mooney. 2000. Automated construction of database interfaces: Intergrating statistical and relational learning for semantic