R ESIDUAL S KILL O PTIMIZATION FOR T EXT- TO -SQL E NSEMBLES
arXiv:2605.21792v1 [cs.CL] 20 May 2026
Jiongli Zhu1*† Haoquan Guan1*† Parjanya Prajakta Prashant1*† Nikki Lijing Kuang2 Seyedeh Baharan Khatami1† Canwen Xu2 Xiaodong Yu2 Yingyu Lin1 Zhewei Yao2 Yuxiong He2‡ Babak Salimi1†‡ 1 University of California, San Diego 2 Snowflake AI Research
A BSTRACT Text-to-SQL ensembles improve over single-candidate generation by drawing multiple SQL candidates and selecting one, but their effectiveness is bounded by Pass@K, the probability that at least one of K candidates is correct. Existing methods source diversity heuristically through stochastic decoding or prompt variants, leaving candidate sets dominated by correlated failures. We present D IV S KILL-SQL, a residual skill optimization framework that builds complementary agentic Textto-SQL ensembles without model fine-tuning: each new skill is optimized on examples the current skill ensemble fails on, provably targeting its marginal contribution to Pass@K. On Spider2-Lite, D IV S KILL-SQL improves selected accuracy by up to +11.1 points on Snowflake and +8.3 on BigQuery over the strongest ensemble baseline, with consistent gains across two base models (Opus4.6 and GPT-5.4). Skills optimized on a single dialect transfer without retraining across dialects (Snowflake, BigQuery, SQLite) and to a different task formulation, such as BIRD-Critic (+2.6 pts). Error diagnostics show up to 3× fewer hallucinated schema references and function calls, indicating that gains come from genuinely reliable complementary skills rather than surface-form variation.
1
Introduction
Text-to-SQL translates natural language questions into executable SQL queries, making relational databases accessible without SQL expertise. While large language models (LLMs) achieve strong results on standard benchmarks [44, 30], real-world queries, which involve large schemas, dialect-specific syntax, and multi-step query logic, remain difficult. This has motivated agentic Text-to-SQL systems that inspect schemas, execute intermediate queries, observe feedback, and iteratively repair errors [35, 9]. Yet even with multi-step interaction, no single agentic execution reliably solves every query, which has motivated ensembling as a complementary axis of robustness [31, 41, 22, 9]: rather than committing to a single SQL generation, ensembles generate multiple SQL candidates and select a final answer. The potential of an ensemble is bounded above by Pass@K, the probability that at least one of K candidates is correct [5]: no selector, however good, can recover an answer that was never generated. Pass@K therefore captures what the generation stage is responsible for, separately from selection. Existing Text-to-SQL ensembles produce candidate diversity by combining stochastic decoding, hand-designed prompt or workflow variants, and in some cases multiple fine-tuned generators [31, 22]. Whether the resulting candidates recover different failure cases, however, is left to chance: nothing in the procedure pushes new candidates to solve examples that earlier ones miss. In agentic systems this is especially fragile: randomness in early planning propagates through long reasoning trajectories, so high-temperature sampling produces noisier variants of the same path rather than complementary solutions, and introduces unstable reasoning, spurious joins, and dialect errors that degrade the candidate pool [32]. Pass@K accordingly plateaus, with little gain from additional candidates. To address this, we introduce D IV S KILL-SQL, a residual skill optimization framework that makes candidate complementarity an explicit optimization target instead of leaving it to heuristics or chance. A skill is a high-level instruction file that * Equal contribution. † ‡
Work done while working at Snowflake AI Research. Co-senior authors.
Residual Skill Optimization for Text-to-SQL Ensembles
Skill Learning Stage Training Questions Schemas : List customers … : How many orders … … : Show products …
Inference Stage
Skill Seed Pool
default decompose explore
1. Diverse Candidate Generation
direct coder adversarial fast repair
Skill-conditioned agent trajectories
1 Select skill seed = default 2
3
Skill
= residual (failed on
“List the top 5 customers by revenue.”
).
:= Optimize on questions D
Evaluate on D := Failed ones
Select skill seed = direct coder
Skill := Optimize on questions
Evaluate on := Failed ones
Select skill seed = decompose
Skill
Evaluate on := Failed ones
:= Optimize on questions
Prefer schema review
Skill Prefer fast SQL drafting
Draft SQL
Execute
Explore schema
Sample rows
Draft SQL
</>
Decompose Draft Compose Question subquery SQL </>
Draft SQL
Execute
Repair SQL
Complementary skills produce less redundant candidates.
2. Pairwise Candidate Selection Candidates
Learned Skills (from diverse seeds)
Skill
Inspect schema
</>
Schemas
…
4
</>
Question
Iterative Skill Optimization Notation: D = training set,
A P REPRINT
Skill Prefer question decomposition
Skill Prefer self-check repair
Pairwise LLM Judge vs
Winner
vs
Winner
Rank by Win Rate Final SQL
… Pairwise ranking is more reliable than direct 1-of-K selection.
New skills target previous failures; the ensemble is complementary.
Figure 1: System diagram of D IV S KILL-SQL. The left panel shows skill construction: starting from diverse strategy prompts, the system repeatedly identifies unsolved questions and refines the next skill toward those remaining cases. The right panel shows test-time execution: multiple skill-guided agents solve the same Text-to-SQL problem through different interaction patterns, producing SQL candidates that are then compared to choose the final output. controls the agent’s decomposition style, schema-exploration policy, drafting strategy, and repair logic; D IV S KILL-SQL operates entirely at this prompt level, with no model fine-tuning. Starting from a base skill, D IV S KILL-SQL evaluates the agent on the training set, identifies the unresolved examples, and uses reflective prompt optimization [1] to refine a new skill specifically on this residual; it repeats round by round until a pool of K complementary skills has been learned. A new skill need not to be globally better than its predecessors; it is useful precisely when it recovers examples they miss. At inference, D IV S KILL-SQL runs each learned skill and selects a final SQL from the resulting candidate set via pairwise comparison [31]. Each skill is trained to cover what the others miss, so the ensemble is complementary by construction rather than by chance. We evaluate D IV S KILL-SQL on recent Text-to-SQL benchmarks across four dialects: Spider2-Lite over SQLite, Snowflake, and BigQuery, and BIRD-Critic over PostgreSQL. D IV S KILL-SQL improves selected accuracy on Spider2Lite by up to +11.1 points on Snowflake and +8.3 on BigQuery over the strongest ensemble baseline, with consistent gains across two base models (Opus-4.6 and GPT-5.4). Skills optimized on standard Text-to-SQL in a single dialect (Snowflake) transfer without retraining to BigQuery and SQLite. Moreover, skills optimized for standard Text-to-SQL generalize to the debugging-style generation setting of BIRD-Critic, where D IV S KILL-SQL improves accuracy by +2.6. Error diagnostics show up to 3× fewer hallucinated schema references and unsupported-function calls, and tool-use trajectory analysis shows that D IV S KILL-SQL reduces redundancy in agent behavior by 19%–28%, producing more diverse schema-inspection, decomposition, drafting, execution, and repair patterns than repeated runs of the same agent. Our contributions are as follows: • We formulate Text-to-SQL ensembling as residual Pass@K optimization over agent skills, where the goal is to learn complementary behaviors that cover different failure modes. • We propose D IV S KILL-SQL, a residual skill optimization framework that improves candidate-set coverage without model fine-tuning or high-temperature sampling. • We show that optimizing each new skill on unresolved examples directly targets its marginal contribution to Pass@K. • We evaluate D IV S KILL-SQL across recent Text-to-SQL benchmarks and four SQL dialects, showing improved accuracy and reduced redundancy in agent trajectories. 2
Residual Skill Optimization for Text-to-SQL Ensembles
2
A P REPRINT
Related Works
Pass@K Optimization Pass@K measures whether a model produces at least one correct solution among K samples [5], and has recently been adopted as an optimization target. Yue et al. [45] show that RL with verifiable rewards improves Pass@1 but not Pass@K: training reduces output diversity and leaves hard problems outside the base model’s support unsolved. This motivates directly optimizing for diversity, which several works pursue through policy optimization [6, 36, 43]. However, these methods all require parameter updates, limiting them to open-weight models and to improving a single model’s output distribution. Our method instead learns K complementary skills without parameter updates, improving Pass@K through ensemble construction for both open and closed models. Skill and Prompt Optimization Prompts strongly influence LLM behavior [3], motivating automatic optimization methods that improve prompts from task feedback rather than manual design. Early work optimizes discrete trigger tokens or textual prompts [33], while recent methods use LLMs to propose, evaluate, and revise prompts iteratively [52, 40, 1]. Boosted prompt ensembles [28] also optimize prompt ensembles by adding few-shot prompts that target examples on which the current ensemble is uncertain or incorrect. Beyond short prompts and few-shot demonstrations, agent systems increasingly use skills: modular instruction files encoding task strategies, constraints, and tool-use policies [47]. Recent work explores agents that rewrite their skills [51], jointly optimize skills with model parameters [38], or evolve and reuse skills over time [24, 48, 49]. Our work follows this direction but targets ensemble construction: rather than learning one strong skill, we learn complementary skills that address different failure modes and improve the ensemble’s Pass@K. Text-to-SQL Early LLM-based Text-to-SQL methods prompt a single model to generate SQL directly [10, 34, 17, 18, 4]. More recent systems use agent-based pipelines that decompose the task into schema pruning, evidence extraction, SQL generation, execution, and refinement [29, 12, 37, 35, 39, 8, 50]. Several works further improve performance through ensembling [31, 9, 41, 22, 15, 13]: CHASE-SQL generates candidates from diverse prompts and selects among them via a tournament [31]; MARS-SQL trains a multi-agent system with RL [41, 23, 46]; and XiYan-SQL fine-tunes multiple generators to induce diversity [22, 42]. In contrast, our method requires no weight modifications or hand-designed ensembles. We optimize skills to explicitly construct complementary generators, increasing Pass@K coverage while remaining applicable to both open and closed LLMs.
3
Method
3.1 Setup and Notation Text-to-SQL with agentic execution. A Text-to-SQL instance is a pair (q, D) consisting of a natural-language question q and a relational database D, and the goal is to produce an executable SQL query whose result on D matches that of a gold reference. We follow the agentic execution paradigm of recent Text-to-SQL systems [35, 9]: rather than generating SQL in a single forward pass, an agent interleaves tool calls, such as inspecting schema, sampling rows, drafting candidate SQL, and repairing errors, over multiple steps before returning a final query. Skills. We modulate the agent’s behavior through skills. A skill s is a high-level instruction file (a system prompt expressed in natural language) that controls the agent’s reasoning and tool-use policy: which decomposition style to favor, when to explore the schema versus draft directly, what repair patterns to apply on execution errors, and so on. We write as for the agent equipped with skill s and treat s as identified with its prompt πs , so that the optimization space S is the space of natural-language instruction files. Two distinct skills induce two genuinely different agent trajectories on the same input, not merely two stochastic samples of the same trajectory. Example 3.1 (Skill Examples). We show two simplified skills below that differ not only in wording but in the agent trajectory they encourage: decompose delays final SQL generation until the query logic has been broken into verified subcomponents, whereas direct_coder pushes the agent to draft early and rely on execution feedback for rapid repair. decompose skill
direct_coder skill
Break complex questions into simple subqueries, build bottom-up. 1. PARSE the question into atomic requirements. 2. BUILD each piece as a standalone CTE. 3. COMPOSE CTEs into the final query using WITH...SELECT.
You are an EFFICIENT SQL writer. Write SQL quickly, test, iterate. 1. Read the question carefully. Identify the core tables, joins, and aggregations. 2. Write your best SQL attempt IMMEDIATELY based on the schema. 3. Execute it. If errors occur, read the error message carefully and fix it.
Notation. Let X denote the space of input tasks (q, D) and let P be the underlying task distribution. For a skill s ∈ S, we write ps (x) ∈ [0, 1] for the probability that one execution of as on input x ∈ X produces a correct SQL query (i.e., a 3
Residual Skill Optimization for Text-to-SQL Ensembles
A P REPRINT
Algorithm 1 D IV S KILL-SQL: batch-sequential residual skill optimization. (0)
Require: training question pool Dtrain ; seed skill pool S0 = {s1 , . . . , sK } with prompts {πs }s∈S0 ; batch size b; number of batches T ; skill optimizer SkillOptimizer(π, R) that returns a refined prompt. (0) 1: πs ← πs for each s ∈ S0 ▷ current seed pool 2: for t = 1, . . . , T do 3: sample a batch Bt ⊆ Dtrain with |Bt | = b ▷ draw from question pool 4: Rt,0 ← Bt ▷ initial residual 5: choose an ordering σt : [K] → S0 of the seed pool ▷ without replacement 6: for j = 1, . . . , K do 7: s ← σt (j) ▷ draw next seed 8: π es ← SkillOptimizer(πs ; Rt,j−1 ) ▷ optimize on residual 9: if p̂πes (Rt,j−1 ) > p̂πs (Rt,j−1 ) then 10: πs ← π es ▷ accept update on the residual 11: end if 12: Rt,j ← { x ∈ Rt,j−1 : as fails on x } ▷ update residual with new πs 13: end for (t) 14: πs ← πs for each s ∈ S0 ▷ commit seed-pool update 15: end for (T ) 16: return πs s∈S 0
query whose execution result matches the gold reference). For a finite training set Dtrain ⊆ X and a subset R ⊆ Dtrain , we write X 1 p̂s (R) = |R| ps (x) x∈R
for the empirical success rate of skill s on R. For a collection of K skills A = {s1 , . . . , sK }, the population Pass@K, defined as the probability that at least one of the K corresponding agent executions succeeds, is YK Pass@K(A) = Ex∼P 1 − 1 − psj (x) . j=1
3.2 Residual Skill Optimization The residual principle. As demonstrated in Figure 1, we construct an ensemble of K skills sequentially. After selecting skills s1 , . . . , sj−1 , we define the residual training set Rj−1 = xi ∈ Dtrain : asℓ fails on xi ∀ℓ < j , and pick the next skill by maximizing success on this residual: sj ∈ arg maxs∈S p̂s (Rj−1 ). Later skills are therefore not pushed to be globally better than earlier ones. They are pushed to cover examples the current ensemble misses—which is exactly the marginal contribution of a new skill to Pass@K. This is the mechanism by which the procedure encourages complementary skills and directly targets ensemble coverage rather than average accuracy. The D IV S KILL-SQL algorithm. The residual arg-max above is an idealized objective: it assumes access to the full training distribution and optimization over the infinite space of natural-language skills. D IV S KILL-SQL turns this principle into a practical batch-sequential process presented in Algorithm 1. At each of T rounds, D IV S KILL-SQL draws a fresh batch Bt ⊆ Dtrain and performs one pass of the residual principle over the seed pool: skills are drawn from S0 in randomized ordering; after each per-skill SkillOptimizer call, examples newly solved by the updated skill are removed from the residual set; and at the end of the batch, the accepted prompt updates are committed back to the seed pool, thus the pool evolves from batch to batch. The algorithm has two main ingredients that make the ideal residual arg-max practical: a finite set of diverse seed skills that defines the initial search space, and an inner-loop optimizer that refines each seed on the current residual failures. Skill seed pool. The arg-max in arg maxs∈S p̂s (Rj−1 ) is over an infinite, unstructured space of natural-language prompts and is intractable directly. To make the inner optimization tractable, we initialize from a small seed pool (0) (0) S0 = {s1 , . . . , sK } of K LLM-assisted, manually curated skills, each encoding a distinct high-level reasoning strategy. Concretely, we first run a generic LLM agent on a subset of the training data and inspect representative and recurring failure modes, and propose diverse strategy prompts from several perspectives: whether to decompose the question before coding, how much schema and value exploration to perform before drafting SQL, etc. We inspect and keep only strategies with distinct intended trajectories, removing near-duplicates that differ only in wording. 4
Residual Skill Optimization for Text-to-SQL Ensembles
A P REPRINT
Example 3.1 presents two simple, high-level skills we adopted as seeds, and the comprehensive list of seed skills is provided in Appendix C. Subsequent rounds optimize prompts starting from the seed pool rather than from scratch, restricting the effective search space to natural refinements of these high-level strategies. Inner-loop optimization. The inner loop uses an LLM to propose refined skill prompts based on observed agent failures, a standard reflective prompt-optimization step [11, 52, 40, 1]. At each round, the LLM is shown failure traces from running the current skill prompt πs on the residual set Rt,j−1 , proposes a refined prompt π es that addresses those failures, and the refinement is accepted if it improves recovery on Rt,j−1 . The agentic setting introduces non-trivial structure: each failure trace contains a full sequence of schema inspections, intermediate query executions, and repair attempts, providing rich diagnostic signal but also requiring the optimizer to attribute blame across multi-step trajectories rather than to a single output. The D IV S KILL-SQL-specific element is the choice of training subset: by optimizing each skill against the residual rather than the full training set, refinements are biased toward recovering prior-ensemble failures rather than improving average accuracy on already-solved examples. The following example of the decompose skill evolution illustrates how residual optimization turns diversity in the seed pool into more targeted coverage in the inner loop optimization. Example 3.2 (Decompose Skill Evolution). The decompose seed skill in Example 3.1 already differs from skills such as direct_coder: it encourages the agent to solve a query by breaking it into subproblems rather than drafting SQL immediately and repairing it based on execution feedback. However, the seed remains too high-level to specify what the agent should verify during decomposition. When optimized on the residual set, the prompt is refined precisely around the frequent kinds of errors that other skills leave unresolved. The example below shows the optimized decompose skill. optimized decompose skill Break complex questions into simple subqueries, build bottom-up. Step 1: PARSE the question into atomic requirements - What is the output? columns, derived metrics, ratios, counts, sums. - What is the grain? one row per what? month? patient? group? - What grouping dimensions are needed? only those that match the desired grain. - Is there ranking, ordering, limiting, or a ratio/composition calculation? Step 2: ANCHOR the grain before grouping - Match GROUP BY columns precisely to the output grain: no more, no less. - If the question asks for monthly totals, group by month only; do not add route, city, or other columns unless explicitly requested. - If the question asks for a split, identify the correct column and values that represent that split; do not substitute a loosely related column. Step 3: BUILD each piece as a standalone CTE - Add joins one at a time, verifying row counts do not explode. - For ratio/composition queries, compute totals in one CTE, subgroup counts in another, then JOIN and divide. Step 4: VERIFY join logic and filter semantics - Confirm join keys actually link the intended entities; avoid fan-out. - Confirm filter values match the domain as they appear in the data.
After optimization, it evolves from generic advice to “build CTEs” into a concrete rubric: identify the intended output grain before grouping, match GROUP BY columns to that grain, choose the correct column for requested splits, and compute ratio/composition queries using compatible aggregation levels. Thus, the learned skill is not merely a more detailed version of the seed prompt; its added details are shaped by the residual cases it is meant to cover. The full before/after skill comparison pool is provided in Appendix C. In practice, we further improve the finite-batch learning procedure through careful reflection prompt and reward design, rotating the skill order, etc.; Appendix B.2 discusses these implementation practices in detail. Population-level guarantee. Under the population-level objective, residual skill optimization is greedy maximization of the Pass@K coverage objective. Formally, Proposition A.1 (stated and proved in Appendix A.1) shows that, in the population limit, Pass@K({s1 , . . . , sK }) ≥ (1 − 1/e) max Pass@K(A). |A|≤K
Thus, the learned skill bank is guaranteed to achieve at least a constant-factor fraction of the best possible K-skill ensemble under the population objective. The intuition is that the marginal value of a new skill lies in its ability to solve problems that the current skill set still fails to address. Residual optimization therefore greedily adds the skill with the largest additional coverage of the remaining failure, yielding the standard approximation guarantee for monotone submodular maximization [25]. 5
Residual Skill Optimization for Text-to-SQL Ensembles Seed skill
Core idea
default explore_heavy direct_coder decompose conservative adversarial_checker template_first fast_error_repair
Balanced exploration and testing Thorough data profiling before drafting Quick draft, and refine incrementally Build from validated substeps like joins Use the simplest faithful query Stress-test joins, filters, grouping, etc. Start from templates and adapt Repair concrete errors one at a time
A P REPRINT
Table 1: Summary of initial seed skills. 3.3 Inference At inference time, the K learned skill-conditioned agents run in parallel on each test instance, producing K candidate SQL queries. We select the final query using pairwise candidate comparisons following Pourreza et al. [31]. Since pairwise comparison scales quadratically in K, we first deduplicate candidates by execution output: queries returning identical results on the target database are collapsed into one equivalence class, with one representative retained. This leaves G ≤ K candidates and reduces the number of comparisons when multiple skills agree. We then1 run an exhaustive round-robin over all G 2 unordered pairs, rather than sampling pairs, because G is small in practice. To mitigate LLM judge position bias, each pair (i, j) is judged twice with swapped presentation order. Each judgment gives one win to the selected candidate, and we return the candidate with the highest win count, breaking ties arbitrarily.
4
Experiments
In the experiments, we evaluate D IV S KILL-SQL through three research questions: RQ1: End-to-end effectiveness. How does D IV S KILL-SQL compare with state-of-the-art Text-to-SQL systems on recent complex benchmarks? RQ2: Transferability. Do skills optimized on a single dataset or SQL dialect transfer to unseen datasets, alternative dialects, and new task formats? RQ3: Behavioral diversity. How does residual skill optimization change the behavior of an agentic Text-toSQL system, beyond simply changing final SQL strings? 4.1 Experimental Setup Benchmarks. We evaluate D IV S KILL-SQL on two recent Text-to-SQL benchmarks that provide clean ground-truth annotations and involve complex reasoning. First, we use Spider2-Lite [16], which tests complex SQL generation over realistic schemas and multiple dialects. We report results on its SQLite, Snowflake, and BigQuery subsets, with 135, 207, and 209 examples respectively. These subsets differ in schema organization, query style, and dialect-specific syntax, allowing us to study both in-domain performance and cross-dialect transfer. Second, we evaluate on BIRD-Critic [20], a SQL debugging benchmark derived from BIRD [19]. Each instance provides a natural-language issue description, a buggy SQL query, and database context; the system must diagnose and repair the query rather than generate SQL from scratch. This setting tests whether D IV S KILL-SQL transfers beyond direct SQL generation to agentic SQL correction. We evaluate on its pure PostgreSQL version. Skill optimization. We adopt the state-of-the-art prompt and skill optimization technique GEPA [1] for skill optimization. For the Spider2-Lite benchmark, we learn skills from only approximately 200 examples sampled from proprietary data in Snowflake SQL dialect, and then evaluate the learned skills on Spider2-Lite. For the BIRD-Critic benchmark, we optimize skills on BIRD-mini-dev [19], which contains 500 standard Text-to-SQL examples. Baselines. We compare D IV S KILL-SQL against representative open-source or most relevant Text-to-SQL systems, including DIN-SQL [29], R E F O RCE [9], and CHASE-SQL [31]. We use the open-source implementations of DIN-SQL and R E F O RCE directly. Since CHASE-SQL was originally designed for a workflow-based pipeline, to adapt it to our agentic setting, we retain its transferable design choices, including schema-link shuffling, hightemperature sampling, and pairwise candidate selection, while replacing the fixed, manually designed workflow with our agent architecture2 . By default, we use the Opus-4.6[2] model. However, we also show results with GPT-5.4 [27]. We share the implementation details in Section B. Metrics. We report three metrics. Pass@1 measures the execution accuracy of a single generated candidate. Pass@8 measures the oracle candidate-set accuracy: a problem is counted as solved if at least one of the eight generated 1
With K = 8, G ≈ 1.7 for baselines and 2.6 for D IV S KILL-SQL on BIRD-Critic. In our experiments, the workflow-based solution consistently underperforms the agentic solution unless heavy engineering effort is applied, which led us to build around the agentic approach. 2
6
Residual Skill Optimization for Text-to-SQL Ensembles
SQLite
Snowflake
A P REPRINT
BigQuery
Method
pass@1 pass@8 Sel. acc. pass@1 pass@8 Sel. acc. pass@1 pass@8 Sel. acc.
DIN-SQL R E F O RCE CHASE-SQL D IV S KILL-SQL
40.74 55.28 62.59 62.13
/ / 76.30 73.33
/ 57.78 63.70 64.44
0.97 43.47 51.21 60.08
/ / 68.60 72.46
/ 50.72 53.14 64.25
18.54 51.10 59.02 62.07
/ / 71.71 73.17
/ 55.12 56.59 64.88
(a) SQLite
Snowflake
BigQuery
Method
pass@1 pass@8 Sel. acc. pass@1 pass@8 Sel. acc. pass@1 pass@8 Sel. acc.
DIN-SQL R E F O RCE CHASE-SQL D IV S KILL-SQL
37.04 64.44 61.57 62.87
/ / 83.70 84.44
/ 58.52 66.67 71.85
0 47.83 51.69 52.23
/ / 73.91 76.33
/ 41.06 57.97 61.84
2.93 48.29 53.90 57.80
/ / 74.15 77.56
/ 51.22 59.51 63.41
(b)
Table 2: Spider2-Lite results across SQLite, Snowflake, and BigQuery using (a) Opus 4.6 and (b) GPT-5.4. Hallucination Type
CHASE-SQL D IV S KILL-SQL Ratio
Pools with invalid-reference candidate Solvable pools with invalid-reference candidate Invalid-reference candidate slots Missing-function hallucination cases
10 6 19 6
7 2 13 2
1.43x 3.00x 1.46x 3.00x
Table 3: Hallucination diagnostics on Snowflake instances based on invalid-reference failures. candidates is correct. Pass@8 therefore measures the quality and coverage of candidate generation independently of selection3 . Finally, selected accuracy measures the execution accuracy of the single SQL query returned by the selector, either through LLM judge or majority voting, and is the end-to-end performance of the deployed system. This applies to ensemble-based methods including D IV S KILL-SQL, R E F O RCE, and CHASE-SQL. Skill seed pool and choice of K. We observe the Pass@k curve for various methods nearly saturates beyond K=8 empirically. Therefore, unless explicitly specified, we use K=8 by default in the experiments. We initialize the optimization with K=8 seed skills (see seed construction in Section 3), each representing a distinct agent behavior family. Table 1 lists all seeds with a one-line strategy summary; full prompt text is given in Appendix C. 4.2 End-to-End Performance and Transferability Tables 2a, 2b and 5 report end-to-end results on BIRD-Critic and Spider2-Lite, grouped by dialect. Overall, D IV S KILL-SQL achieves the strongest selected accuracy in almost all settings and consistently outperforms the strongest ensemble baseline, CHASE-SQL. The advantage is most visible on the harder dialects in Table 2a. With Opus-4.6, D IV S KILL-SQL improves selected accuracy over CHASE-SQL by +11.11 on Snowflake and +8.29 on BigQuery, where large schemas and dialect-specific syntax make unstable decoding more costly. On SQLite, D IV S KILL-SQL has slightly lower Pass@8 than CHASE-SQL (73.33 vs. 76.30), but still achieves higher selected accuracy (64.44 vs. 63.70). This suggests that raw coverage alone is insufficient: SQLite is easier, so stochastic sampling can already cover many cases, but the additional candidates may include plausible wrong queries that are harder for the selector to distinguish [32]. The same pattern holds with GPT-5.4 in Table 2b, where D IV S KILL-SQL improves both Pass@8 and selected accuracy across all three dialects. Note that R E F O RCE selects its final SQL by majority voting over internal candidates. With GPT-5.4, its selected result underperforms mean Pass@1 on SQLite and Snowflake, showing that majority voting can fail when many candidates converge to the same incorrect result. See Appendix D for detailed analysis. Hallucination and Error Analysis. Tables 3 and 4 provide a more structured error breakdown, evaluated on the Snowflake part of Spider2-Lite. Compared with CHASE-SQL, D IV S KILL-SQL produces fewer invalid-reference failures: the number of pools containing such a candidate drops from 10 to 7, and among solvable pools, the count drops from 6 to 2. Missing-function hallucinations also decrease from 6 to 2. The structural comparison against gold SQL shows a similar pattern: D IV S KILL-SQL makes fewer errors involving DISTINCT, window functions, and UNION 3
We compute Pass@8 based on internal candidates of ensemble-based methods. As R E F O RCE is generating a dynamic number of candidates in its workflow, we are not able to compute Pass@8.
7
Residual Skill Optimization for Text-to-SQL Ensembles
A P REPRINT
Structural Mismatch vs. Gold SQL CHASE-SQL D IV S KILL-SQL Ratio Wrong DISTINCT usage Wrong window-function usage Wrong UNION structure
10 2 3
15 6 6
1.50x 3.00x 2.00x
Table 4: Structural mismatch analysis on Snowflake instances with reference SQL available. 85% 60%
Pass@k
Pass@k
80% 75% 70%
Ours Base Opt. Base Seeds
65% 60% 2
4
6
8
a)
k
10
12
14
55%
45%
16
Ours Base Opt. Base Seeds
50%
2
4
6
8
b)
k
10
12
14
16
Figure 2: Pass@k comparison between D IV S KILL-SQL and its variants on 100-instance subsets of a) Spider2-lite and b) Bird-Critic. structure. These results indicate that residual skills improve diversity in a more controlled way. Rather than relying on high-temperature perturbations that can introduce hallucinated references or unstable SQL structures, D IV S KILL-SQL induces different agent behaviors while preserving candidate quality. Transfer across SQL dialects and task settings. We next analyze whether skills encode reusable problem-solving strategies or merely overfit to one dialect or task. Recall that on Spider2-Lite, the skills are optimized only on Snowflake SQL format data, but are applied to all dialects without further optimization. The end-to-end results in Tables 2a and 2b show that D IV S KILL-SQL outperforms all baselines on SQLite and BigQuery by a large margin. This suggests that residual skill optimization does not merely memorize Snowflake-specific syntax. Instead, as we will show later in Section 4.4, the learned skills capture higher-level strategies for schema exploration, decomposition, query construction, and error checking that transfer across dialects. Although all text-to-SQL systems, datasets, and benchmarks target Method pass@1 pass@8 Sel. acc. accurate SQL generation given a user’s question and a database, their DIN-SQL 30.38 – – settings vary widely. For instance, BIRD-Critic focuses on debugging R E F O RCE 41.50 – 43.21 SQL using feedback and a given buggy SQL query, which is quite CHASE-SQL 44.69 52.26 46.23 different from existing benchmarks. In this case, finding the training D IV S KILL -SQL 46.16 54.53 48.87 data in the same format is challenging. In this experiment, we explore Table 5: Bird-Critic PostgreSQL Results. whether the skills learned on Bird-mini-dev—a regular Text-to-SQL dataset with no feedback or buggy SQL—translate well to Bird-Critic. Results in Table 5 confirm that the skills transfer: D IV S KILL-SQL improves selected accuracy by +2.64 points over CHASE-SQL, with consistent gains in pass@1 (+1.47) and pass@8 (+2.27). The simultaneous pass@8 improvement indicates that the learned skills broaden the candidate pool, even on a task format (debugging from feedback) they were not optimized for. 4.3
Ablation Studies
To understand the effectiveness of each component of D IV S KILL-SQL, including the residual optimization process and the skill pool, we compare D IV S KILL-SQL with its three variants: Base, which repeatedly samples from the original base skill; Opt. Base, which runs GEPA to optimize residuals from only base skills rather than diverse seeds; and Seeds, which uses the unoptimized initial skill seeds. These baselines separate the effect of residual optimization from the effect of simply sampling more times, optimizing one stronger prompt, or using manually diverse initial instructions. Figure 2 studies how candidate-set coverage changes as we increase the number of generated candidates. Repeated sampling from the base skill, i.e. Base, improves Pass@k only gradually, indicating that independent runs of the same skill tend to make correlated mistakes. Optimizing a single base skill improves individual quality, but still leaves many residual failures uncovered. Initial diverse set of seed skills helps compared to Base in some cases (Figure 2b), but might also degrades performance (Figure 2a). In contrast, D IV S KILL-SQL achieves the strongest Pass@k curve across nearly all values of k. The gain is especially meaningful at small and moderate k, where each additional candidate must cover new failure modes to be useful. As a result, to achieve the same coverage as D IV S KILL-SQL’s pass@8, baselines need 3 to 8 additional passes, making D IV S KILL-SQL a more cost-efficient choice. This behavior is exactly what residual skill optimization is designed to 8
Residual Skill Optimization for Text-to-SQL Ensembles
(b) Base
(c) Pairwise Similarity 0.9
explore 0.71 1.00 0.50 0.70 0.69 0.70 0.58 0.71 heavy direct 0.58 0.50 1.00 0.60 0.60 0.59 0.39 0.54 coder conservative 0.82 0.70 0.60 1.00 0.81 0.82 0.55 0.74
run 2 0.84 1.00 0.82 0.76 0.77 0.83 0.84 0.77
0.8
adversarial 0.79 0.69 0.60 0.81 1.00 0.81 0.56 0.75 checker decompose 0.79 0.70 0.59 0.82 0.81 1.00 0.55 0.76
run 5 0.79 0.77 0.82 0.80 1.00 0.82 0.79 0.84
0.6
run 6 0.84 0.83 0.83 0.76 0.82 1.00 0.82 0.82
0.5
template 0.56 0.58 0.39 0.55 0.56 0.55 1.00 0.59 first fast error 0.76 0.71 0.54 0.74 0.75 0.76 0.59 1.00 repair
run 7 0.80 0.84 0.89 0.76 0.79 0.82 1.00 0.76
run 3 0.81 0.82 1.00 0.76 0.82 0.83 0.89 0.80 run 4 0.80 0.76 0.76 1.00 0.80 0.76 0.76 0.73
n1
n2
n3
n4
n5
n6
n7
n8
ru
ru
ru
ru
ru
ru
ru
ru
run 8 0.79 0.77 0.80 0.73 0.84 0.82 0.76 1.00
Trajectory similarity
run 1 1.00 0.84 0.81 0.80 0.79 0.84 0.80 0.79
de fa ex ult p he lore av dir y co ect de co r ns er ad vati ve ve r ch sari ec al ke de r co mp os te mp e lat e fas firs te t r r re or pa ir
default 1.00 0.71 0.58 0.82 0.79 0.79 0.56 0.76
0.7
12
Ours Base
10
Pair Count
(a) Ours
A P REPRINT
8 6 4 2
0.4
0
0.4
0.6
Similarity
0.8
Figure 3: Trajectory comparison between D IV S KILL-SQL and repeated runs of the default skill on the Snowflake part of Spider2-Lite. (b) Base
(c) Pairwise Similarity
explore 0.71 1.00 0.50 0.70 0.70 0.70 0.51 0.69 heavy direct 0.56 0.50 1.00 0.60 0.54 0.56 0.36 0.49 coder conservative 0.79 0.70 0.60 1.00 0.77 0.80 0.52 0.71
run 2 0.79 1.00 0.77 0.85 0.72 0.78 0.75 0.83
0.8
adversarial 0.76 0.70 0.54 0.77 1.00 0.76 0.51 0.72 checker decompose 0.77 0.70 0.56 0.80 0.76 1.00 0.51 0.71
run 5 0.82 0.72 0.81 0.77 1.00 0.81 0.77 0.76
0.6
run 6 0.77 0.78 0.81 0.83 0.81 1.00 0.80 0.80
0.5
template 0.51 0.51 0.36 0.52 0.51 0.51 1.00 0.55 first fast error 0.72 0.69 0.49 0.71 0.72 0.71 0.55 1.00 repair
run 7 0.77 0.75 0.79 0.78 0.77 0.80 1.00 0.73
run 3 0.77 0.77 1.00 0.80 0.81 0.81 0.79 0.74 run 4 0.85 0.85 0.80 1.00 0.77 0.83 0.78 0.76
8 6 4 2
0.4
0
0.4
n8
n7
0.7
Ours Base
10
ru
n6
ru
ru
n5
n4
ru
n3
ru
ru
n2 ru
ru
n1
run 8 0.77 0.83 0.74 0.76 0.76 0.80 0.73 1.00
Trajectory similarity
0.9
de fa ex ult p he lore av dir y co ect de co r ns er ad vati ve ve r ch sari ec al ke de r co mp os te mp e lat e fas firs te t r r re or pa ir
run 1 1.00 0.79 0.77 0.85 0.82 0.77 0.77 0.77
Pair Count
(a) Ours default 1.00 0.71 0.56 0.79 0.76 0.77 0.51 0.72
0.6
Similarity
0.8
Figure 4: Trajectory comparison between D IV S KILL-SQL and repeated runs of the default skill on the BigQuery part of Spider2-Lite. (b) Base
(c) Pairwise Similarity
explore 0.72 1.00 0.46 0.71 0.70 0.70 0.51 0.70 heavy direct 0.48 0.46 1.00 0.57 0.50 0.54 0.34 0.51 coder conservative 0.77 0.71 0.57 1.00 0.77 0.79 0.54 0.79
run 2 0.80 1.00 0.79 0.73 0.82 0.77 0.75 0.81
adversarial 0.78 0.70 0.50 0.77 1.00 0.76 0.54 0.72 checker decompose 0.76 0.70 0.54 0.79 0.76 1.00 0.53 0.74
run 5 0.77 0.82 0.77 0.83 1.00 0.83 0.84 0.83
template 0.52 0.51 0.34 0.54 0.54 0.53 1.00 0.58 first fast error 0.73 0.70 0.51 0.79 0.72 0.74 0.58 1.00 repair
run 7 0.76 0.75 0.78 0.82 0.84 0.80 1.00 0.85
0.4
run 8 0.79 0.81 0.81 0.79 0.83 0.80 0.85 1.00
0.3
Trajectory similarity
0.7
run 4 0.78 0.73 0.80 1.00 0.83 0.80 0.82 0.79
0.6
n8 ru
n7 ru
n6 ru
n5 ru
n4 ru
n3 ru
n2 ru
n1
run 6 0.78 0.77 0.77 0.80 0.83 1.00 0.80 0.80
0.5
Ours Base
15.0
0.8
run 3 0.78 0.79 1.00 0.80 0.77 0.77 0.78 0.81
ru
de fa ex ult p he lore av dir y co ect de co r ns e ad rvat ve ive r ch sari ec al ke de r co mp os te mp e lat e fas firs te t re rror pa ir
run 1 1.00 0.80 0.78 0.78 0.77 0.78 0.76 0.79
12.5
Pair Count
(a) Ours default 1.00 0.72 0.48 0.77 0.78 0.76 0.52 0.73
10.0 7.5 5.0 2.5 0.0
0.4
0.6
Similarity
0.8
Figure 5: Trajectory comparison between D IV S KILL-SQL and repeated runs of the default skill on the SQLite part of Spider2-Lite. produce: each later skill is not optimized to be globally better on all questions, but to solve examples that previous skills miss. 4.4 How Diverse Skills Change Agent Behavior To understand why residual skill optimization changes Text-to-SQL agent behavior, we analyze agent trajectories rather than only final SQL outputs. A trajectory records the sequence of high-level actions taken by an agent, such as schema inspection, exploratory SQL generation, and error repair. We measure trajectory dissimilarity between two runs using edit distance normalized to [0, 1], and define trajectory similarity as 1 minus this normalized distance. Lower similarity therefore indicates that two candidates are produced through more distinct reasoning and tool-use paths. Figure 3 shows a clear contrast between D IV S KILL-SQL and repeated sampling. Repeated runs form a high-similarity cluster, with most pairwise similarities concentrated around 0.75–0.85, suggesting that sampling alone mostly produces variants of the same reasoning path. In contrast, D IV S KILL-SQL spreads trajectories over a much wider similarity range: learned skills such as direct-coder, template-first, decomposition, and exploration-heavy form low-similarity pairs, indicating that they induce genuinely different agent behaviors rather than merely different SQL surface forms. 9
Residual Skill Optimization for Text-to-SQL Ensembles
A P REPRINT
This pattern also transfers to the BigQuery and SQLite subsets of Spider2-Lite (Figures 4 and 5), where skills learned from Snowflake data are applied without retraining, suggesting that the behavioral changes are not tied to a single SQL dialect. Overall, the analysis supports the central mechanism of D IV S KILL-SQL: learned skills change how the agent approaches the task, thereby producing candidate sets with less-correlated failure modes and helping explain the stronger Pass@K curve in Figure 2.
5
Discussion, Limitations, and Conclusions
Discussion and limitations. D IV S KILL-SQL primarily improves candidate generation and does not focus on enhanced candidate selection method. Across several settings, there remains a substantial gap between Pass@8 and selected accuracy, indicating that the correct SQL is often present in the candidate pool but not selected. Our pairwise comparison procedure mitigates direct 1-of-K selection difficulty and reduces position bias by swapping candidate order, but it still relies on an LLM judge and scales quadratically in the number of surviving candidates. Future work could combine residual skill optimization with stronger selectors. Recent advances in language model reasoning demonstrate that verifiers trained on self-generated correct and incorrect trajectories—often formulated as outcome or process reward models—can significantly enhance test-time selection among multiple candidates [7, 21, 14, 26]. This paradigm is naturally aligned with our framework: D IV S KILL-SQL already generates diverse candidate reasoning paths and SQL programs, while execution feedback provides an abundant, automated source of positive and negative supervision to effectively train such selectors without human annotation. In addition, our evaluation focuses on recent Text-to-SQL and SQL-debugging benchmarks with executable ground truth. Although the learned skills transfer across SQL dialects and even from standard Text-to-SQL training data to BIRD-Critic, broader deployment settings may introduce additional challenges, including ambiguous user intent, missing schema documentation, and multi-turn interactive feedback-based debugging. Future work might explore how to adapt D IV S KILL-SQL to these more complex settings. Conclusions. We introduce D IV S KILL-SQL, a residual skill optimization framework for constructing complementary Text-to-SQL ensembles. Rather than relying on hand-designed prompt or workflow variants, or high-temperature sampling, D IV S KILL-SQL learns a bank of skill-conditioned agents, where each skill is optimized to recover examples missed by the others. This directly targets candidate-set coverage and improves Pass@K while preserving the quality of SQL candidates. Across Spider2-Lite and BIRD-Critic, spanning multiple SQL dialects and task formats, D IV S KILL-SQL improves end-to-end accuracy over strong ensemble baselines while producing fewer hallucinated outputs and less redundant agent behavior.
References [1] Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, et al. Gepa: Reflective prompt evolution can outperform reinforcement learning. arXiv preprint arXiv:2507.19457, 2025. [2] Anthropic. Introducing Claude Opus 4.6. https://www.anthropic.com/news/claude-opus-4-6, February 2026. [3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. [4] Kaiwen Chen, Yueting Chen, Nick Koudas, and Xiaohui Yu. Reliable text-to-sql with adaptive abstention. Proc. ACM Manag. Data, 3(1), February 2025. doi: 10.1145/3709719. URL https://doi.org/10.1145/3709719. [5] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. [6] Zhipeng Chen, Xiaobo Qin, Youbin Wu, Yue Ling, Qinghao Ye, Wayne Xin Zhao, and Guang Shi. Pass@ k training for adaptively balancing exploration and exploitation of large reasoning models. arXiv preprint arXiv:2508.10751, 2025. [7] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. 10
Residual Skill Optimization for Text-to-SQL Ensembles
A P REPRINT
[8] Yaxun Dai, Haiqin Yang, Mou Hao, and Pingfu Chao. PARSQL: Enhancing text-to-SQL through SQL parsing and reasoning. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Findings of the Association for Computational Linguistics: ACL 2025, pages 661–681, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-256-5. doi: 10.18653/v1/2025.findings-acl.37. URL https://aclanthology.org/2025.findings-acl.37/. [9] Minghang Deng, Ashwin Ramachandran, Canwen Xu, Lanxiang Hu, Zhewei Yao, Anupam Datta, and Hao Zhang. Reforce: A text-to-sql agent with self-refinement, format restriction, and column exploration. In ICLR 2025 Workshop: VerifAI: AI Verification in the Wild, 2025. [10] Xuemei Dong, Chao Zhang, Yuhang Ge, Yuren Mao, Yunjun Gao, Jinshu Lin, Dongfang Lou, et al. C3: Zero-shot text-to-sql with chatgpt. arXiv preprint arXiv:2307.07306, 2023. [11] Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. Promptbreeder: Self-referential self-improvement via prompt evolution. arXiv preprint arXiv:2309.16797, 2023. [12] Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. Text-to-sql empowered by large language models: A benchmark evaluation. arXiv preprint arXiv:2308.15363, 2023. [13] Yu Guo, Dong Jin, Shenghao Ye, Shuangwu Chen, Jian Yang, and Xiaobin Tan. SQLForge: Synthesizing reliable and diverse data to enhance text-to-SQL reasoning in LLMs. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Findings of the Association for Computational Linguistics: ACL 2025, pages 8441–8452, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-2565. doi: 10.18653/v1/2025.findings-acl.443. URL https://aclanthology.org/2025.findings-acl.443/. [14] Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. V-star: Training verifiers for self-taught reasoners. arXiv preprint arXiv:2402.06457, 2024. [15] Dongjun Lee, Choongwon Park, Jaehyuk Kim, and Heesoo Park. MCS-SQL: Leveraging multiple prompts and multiple-choice selection for text-to-SQL generation. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert, editors, Proceedings of the 31st International Conference on Computational Linguistics, pages 337–353, Abu Dhabi, UAE, January 2025. Association for Computational Linguistics. URL https://aclanthology.org/2025.coling-main.24/. [16] Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin Su, Zhaoqing Suo, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, et al. Spider 2.0: Evaluating language models on real-world enterprise text-to-sql workflows. arXiv preprint arXiv:2411.07763, 2024. [17] Boyan Li, Yuyu Luo, Chengliang Chai, Guoliang Li, and Nan Tang. The dawn of natural language to sql: Are we fully ready? Proc. VLDB Endow., 17(11):3318–3331, July 2024. ISSN 2150-8097. doi: 10.14778/3681954. 3682003. URL https://doi.org/10.14778/3681954.3682003. [18] Haoyang Li, Jing Zhang, Hanbing Liu, Ju Fan, Xiaokang Zhang, Jun Zhu, Renjie Wei, Hongyan Pan, Cuiping Li, and Hong Chen. Codes: Towards building open-source language models for text-to-sql. Proc. ACM Manag. Data, 2(3), May 2024. doi: 10.1145/3654930. URL https://doi.org/10.1145/3654930. [19] Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, et al. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. Advances in Neural Information Processing Systems, 36, 2024. [20] Jinyang Li, Xiaolong Li, Ge Qu, Per Jacobsson, Bowen Qin, Binyuan Hui, Shuzheng Si, Nan Huo, Xiaohan Xu, Yue Zhang, et al. Swe-sql: Illuminating llm pathways to solve user sql issues in real-world applications. arXiv preprint arXiv:2506.18951, 2025. [21] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In The twelfth international conference on learning representations, 2023. [22] Yifu Liu, Yin Zhu, Yingqi Gao, Zhiling Luo, Xiaoxia Li, Xiaorong Shi, Yuntao Hong, Jinyang Gao, Yu Li, Bolin Ding, et al. Xiyan-sql: A novel multi-generator framework for text-to-sql. IEEE Transactions on Knowledge and Data Engineering, 2026. 11
Residual Skill Optimization for Text-to-SQL Ensembles
A P REPRINT
[23] Peixian MA, Xialie Zhuang, Chengjin Xu, Xuhui Jiang, Ran Chen, and Jian Guo. SQL-r1: Training natural language to SQL reasoning model by reinforcement learning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026. URL https://openreview.net/forum?id=hgJQcuDwm1. [24] Ziyu Ma, Shidong Yang, Yuxiang Ji, Xucong Wang, Yong Wang, Yiming Hu, Tongwen Huang, and Xiangxiang Chu. Skillclaw: Let skills evolve collectively with agentic evolver. arXiv preprint arXiv:2604.08377, 2026. [25] George L Nemhauser, Laurence A Wolsey, and Marshall L Fisher. An analysis of approximations for maximizing submodular set functions—i. Mathematical programming, 14(1):265–294, 1978. [26] Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen-tau Yih, Sida Wang, and Xi Victoria Lin. Lever: Learning to verify language-to-code generation with execution. In International Conference on Machine Learning, pages 26106–26128. PMLR, 2023. [27] OpenAI. Introducing GPT-5.4. https://openai.com/index/introducing-gpt-5-4/, March 2026. [28] Silviu Pitis, Michael R Zhang, Andrew Wang, and Jimmy Ba. Boosted prompt ensembles for large language models. arXiv preprint arXiv:2304.05970, 2023. [29] Mohammadreza Pourreza and Davood Rafiei. Din-sql: Decomposed in-context learning of text-to-sql with self-correction. Advances in neural information processing systems, 36:36339–36348, 2023. [30] Mohammadreza Pourreza and Davood Rafiei. Evaluating cross-domain text-to-sql models and benchmarks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1601–1611, 2023. [31] Mohammadreza Pourreza, Hailong Li, Ruoxi Sun, Yeounoh Chung, Shayan Talaei, Gaurav Tarlok Kakkar, Yu Gan, Amin Saberi, Fatma Ozcan, and Sercan O Arik. Chase-sql: Multi-path reasoning and preference optimized candidate selection in text-to-sql. arXiv preprint arXiv:2410.01943, 2024. [32] Mohammadreza Pourreza, Shayan Talaei, Ruoxi Sun, Xingchen Wan, Hailong Li, Azalia Mirhoseini, Amin Saberi, Sercan Arik, et al. Reasoning-sql: Reinforcement learning with sql tailored partial rewards for reasoning-enhanced text-to-sql. arXiv preprint arXiv:2503.23157, 2025. [33] Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pages 4222–4235, 2020. [34] Chang-Yu Tai, Ziru Chen, Tianshu Zhang, Xiang Deng, and Huan Sun. Exploring chain of thought style prompting for text-to-sql. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5376–5393, 2023. [35] Shayan Talaei, Mohammadreza Pourreza, Yu-Chen Chang, Azalia Mirhoseini, and Amin Saberi. Chess: Contextual harnessing for efficient sql synthesis. arXiv preprint arXiv:2405.16755, 2024. [36] Christian Walder and Deep Karkhanis. Pass@ k policy optimization: Solving harder reinforcement learning problems. arXiv preprint arXiv:2505.15201, 2025. [37] Bing Wang, Changyu Ren, Jian Yang, Xinnian Liang, Jiaqi Bai, Linzheng Chai, Zhao Yan, Qian-Wen Zhang, Di Yin, Xing Sun, et al. Mac-sql: A multi-agent collaborative framework for text-to-sql. In Proceedings of the 31st International Conference on Computational Linguistics, pages 540–557, 2025. [38] Peng Xia, Jianwen Chen, Hanyang Wang, Jiaqi Liu, Kaide Zeng, Yu Wang, Siwei Han, Yiyang Zhou, Xujiang Zhao, Haifeng Chen, et al. Skillrl: Evolving agents via recursive skill-augmented reinforcement learning. arXiv preprint arXiv:2602.08234, 2026. [39] Yuanzhen Xie, Xinzhou Jin, Tao Xie, Mingxiong Lin, Liang Chen, Chenyun Yu, Lei Cheng, Chengxiang Zhuo, Bo Hu, and Zang Li. Decomposition for enhancing attention: Improving LLM-based text-to-SQL through workflow paradigm. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 10796–10816, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl.641. URL https://aclanthology.org/2024. findings-acl.641/. [40] Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. In The Twelfth International Conference on Learning Representations, 2023. 12
Residual Skill Optimization for Text-to-SQL Ensembles
A P REPRINT
[41] Haolin Yang, Jipeng Zhang, Zhitao He, and Yi R Fung. Mars-sql: A multi-agent reinforcement learning framework for text-to-sql. arXiv preprint arXiv:2511.01008, 2025. [42] Jiaxi Yang, Binyuan Hui, Min Yang, Jian Yang, Junyang Lin, and Chang Zhou. Synthesizing text-to-SQL data from weak and strong LLMs. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7864–7875, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.425. URL https://aclanthology.org/2024.acl-long.425/. [43] Jian Yao, Ran Cheng, Xingyu Wu, Jibin Wu, and Kay Chen Tan. Diversity-aware policy optimization for large language model reasoning. arXiv preprint arXiv:2505.23433, 2025. [44] Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 3911–3921, 2018. [45] Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837, 2025. [46] Bohan Zhai, Canwen Xu, Yuxiong He, and Zhewei Yao. Optimizing reasoning for text-to-SQL with execution feedback. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Findings of the Association for Computational Linguistics: ACL 2025, pages 19206–19218, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-256-5. doi: 10.18653/v1/2025.findings-acl.982. URL https://aclanthology.org/2025.findings-acl.982/. [47] Barry Zhang, Keith Lazuka, and Mahesh Murag. Equipping agents for the real world with agent skills, october 2025. URL https://www. anthropic. com/engineering/equipping-agents-for-the-real-world-with-agent-skills. Accessed, pages 01–28, 2026. [48] Hanrong Zhang, Shicheng Fan, Henry Peng Zou, Yankai Chen, Zhenting Wang, Jiayu Zhou, Chengze Li, WeiChieh Huang, Yifei Yao, Kening Zheng, et al. Evoskills: Self-evolving agent skills via co-evolutionary verification. arXiv preprint arXiv:2604.01687, 2026. [49] Haozhen Zhang, Quanyu Long, Jianzhu Bao, Tao Feng, Weizhi Zhang, Haodong Yue, and Wenya Wang. Memskill: Learning and evolving memory skills for self-evolving agents. arXiv preprint arXiv:2602.02474, 2026. [50] Qinggang Zhang, Hao Chen, Junnan Dong, Shengyuan Chen, Feiran Huang, and Xiao Huang. Structure-guided large language models for text-to-SQL generation. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=gT8JSEFqaS. [51] Huichi Zhou, Siyuan Guo, Anjie Liu, Zhongwei Yu, Ziqin Gong, Bowen Zhao, Zhixun Chen, Menglong Zhang, Yihang Chen, Jinsong Li, et al. Memento-skills: Let agents design agents. arXiv preprint arXiv:2603.18743, 2026. [52] Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In The eleventh international conference on learning representations, 2022.
13
Residual Skill Optimization for Text-to-SQL Ensembles
A
Proofs
A.1
Proof of Proposition A.1
A P REPRINT
We analyze the population-level version of residual skill optimization. In this setting, the next skill is chosen to maximize its expected contribution on the residual failure mass of the current skill bank. The finite-batch procedure in Algorithm 1 can be viewed as an empirical approximation to this objective; deriving finite-sample guarantees would require additional assumptions on sample size and generalization of the learned skills. Proposition A.1 (Residual skill optimization approximates optimal Pass@K). Let S be a fixed skill family, and let ps (x) ∈ [0, 1] denote the probability that one execution of skill s solves input x. For any skill bank A ⊆ S, define its population Pass@K objective as Y F (A) = Ex∼P 1 − (1 − ps (x)) . s∈A
Starting from A0 = ∅, suppose that at each round j = 1, . . . , K, residual skill optimization selects Y sj ∈ arg max Ex∼P ps (x) (1 − ps′ (x)) , s∈S
s′ ∈Aj−1
where Aj−1 = {s1 , . . . , sj−1 }, and sets Aj = Aj−1 ∪ {sj }. Let A⋆ ∈ arg
max
A⊆S, |A|≤K
F (A)
be the optimal size-K skill bank. Then F (AK ) ≥ (1 − 1/e)F (A⋆ ). Equivalently, Pass@K({s1 , . . . , sK }) ≥ (1 − 1/e) max Pass@K(A). |A|≤K
Proof. The proof follows the standard greedy analysis for monotone submodular maximization under a cardinality constraint [25]; we first verify that the Pass@K objective in our setting is indeed monotone submodular. We first show that the Pass@K objective is monotone submodular. For a skill bank A ⊆ S, recall that Y F (A) = Ex∼P 1 − (1 − ps (x)) . s∈A
For any skill s ∈ / A, the marginal gain of adding s is ∆(s | A) = F (A ∪ {s}) − F (A). Expanding the definition of F , we obtain
∆(s | A) = Ex∼P 1 − (1 − ps (x))
Y
(1 − ps′ (x)) − Ex∼P 1 −
s′ ∈A
Y
(1 − ps′ (x))
s′ ∈A
= Ex∼P ps (x)
Y
(1 − ps′ (x)) .
s′ ∈A
Since ps (x) ∈ [0, 1], this marginal gain is nonnegative. Hence F is monotone. Next, let A ⊆ B ⊆ S. Since each factor 1 − ps′ (x) ∈ [0, 1], we have Y Y (1 − ps′ (x)) ≤ (1 − ps′ (x)). s′ ∈B
s′ ∈A
Therefore, ∆(s | B) = Ex∼P ps (x)
Y
(1 − ps′ (x)) ≤ Ex∼P ps (x)
s′ ∈B
Y s′ ∈A
14
(1 − ps′ (x)) = ∆(s | A).
Residual Skill Optimization for Text-to-SQL Ensembles
A P REPRINT
Thus F satisfies diminishing marginal returns and is submodular. The residual arg-max rule is exactly greedy maximization of this objective, because the marginal gain of adding skill s to the current bank Aj−1 is Y ∆(s | Aj−1 ) = Ex∼P ps (x) (1 − ps′ (x)) . s′ ∈Aj−1
Thus choosing sj by residual maximization is the same as choosing the skill with largest marginal increase in F . Let A⋆ ∈ arg max|A|≤K F (A) be an optimal size-K skill bank. Since F is monotone, F (A⋆ ) − F (Aj ) ≤ F (Aj ∪ A⋆ ) − F (Aj ). By submodularity, F (Aj ∪ A⋆ ) − F (Aj ) ≤
X
∆(s | Aj ).
s∈A⋆
Since |A⋆ | ≤ K, at least one skill s ∈ A⋆ has marginal gain at least F (A⋆ ) − F (Aj ) . K Greedy chooses the skill with largest marginal gain, so F (A⋆ ) − F (Aj ) . F (Aj+1 ) − F (Aj ) ≥ K Equivalently, 1 ⋆ F (A ) − F (Aj+1 ) ≤ 1 − F (A⋆ ) − F (Aj ) . K Applying this recurrence for K rounds gives K 1 F (A⋆ ) − F (AK ) ≤ 1 − F (A⋆ ). K Hence K ! 1 F (AK ) ≥ 1 − 1 − F (A⋆ ) ≥ (1 − 1/e)F (A⋆ ). K This proves the claim.
B
Implementation Details and Best Empirical Practices
B.1
Implementation Details
Machine setup. All experiments were run on a single Apple Silicon MacBook Pro (Apple M-series CPU, 10+ cores; 32 GB unified memory; macOS 14). Since our pipeline issues all language-model calls to hosted inference APIs, no local GPU is required: the workstation only orchestrates prompting, parsing, and result aggregation. LLM backbone and inference settings. We evaluate D IV S KILL-SQL with two LLM backbones: Opus-4.6 and GPT-5.4. Both are used in non-reasoning mode with a decoding temperature of 0.2, a canonical choice for coding tasks, and a maximum completion budget of 64000 tokens. The low temperature is a deliberate design choice: since D IV S KILL-SQL achieves diversity through learned skills rather than stochastic decoding, a low temperature preserves the reasoning stability and precision needed for complex SQL generation. Each agent run is allowed up to 12 reasoning turns and 20 SQL executions. CHASE-SQL adaptation. CHASE-SQL was originally designed for a workflow-based Text-to-SQL pipeline. To produce a fair comparison in our agentic setting, we adapt its three transferable design choices to our agent architecture: • Schema-link shuffling. We permute the column ordering presented to the agent across different candidates, matching CHASE-SQL’s schema perturbation strategy. • High-temperature decoding. We set the agent’s decoding temperature to 1.0 for CHASE-SQL candidates, reproducing the stochastic variation that CHASE-SQL uses to induce candidate diversity. • Pairwise candidate selection. We use the same LLM-based pairwise comparison selector for both CHASE-SQL and D IV S KILL-SQL, ensuring the selection mechanism is held constant across methods. 15
Residual Skill Optimization for Text-to-SQL Ensembles
Hyperparameter
Value
Agent (evaluation) Agent model Max agent turns Max SQL executions Max completion tokens
GPT-5.4 (T =0.2) or Sonnet-4.6 12 20 64000
GEPA optimizer Reflection model Task proposal temperature Reflection max tokens Metric calls per skill per batch Reflection minibatch size
GPT-5.4 (T =1.0) or Opus-4.6 0.2 64000 20 3
Training loop Number of batches Batch size (train / val) Held-out validation size Max prompt length
3 70 / 30 80 12000 chars
A P REPRINT
Table 6: GEPA hyperparameters for BIRD-mini-dev skill optimization. Spider2-Lite Snowflake training uses the same settings except the training data source.
Candidate selection. For both D IV S KILL-SQL and CHASE-SQL, we use an LLM-based pairwise selector that compares each candidate pair and selects the winner via win-rate-based aggregation. The selector uses the same backbone model as the candidate generator (either Opus-4.6 or GPT-5.4) at a temperature of 0.2. Each pairwise comparison receives the question, database schema, and the two candidate SQL queries along with execution previews, and returns a preference judgment. GEPA optimization settings. Agent tools.
Table 6 summarizes the GEPA hyperparameters used for skill optimization.
The agent has access to six tools during both training and evaluation:
• execute_sql: run SQL against the live database and return results or error messages. • lookup_docs: retrieve dialect-specific documentation (e.g., dialect function/grammar reference), database meta-data, external knowledge, etc. • review_sql: invoke an LLM-based critic to review the current SQL draft before submission. • get_sql_pattern: retrieve anonymized SQL patterns for similar query types (e.g., top-N, running totals). • get_sql_templates: retrieve SQL templates that are masked training instances categorized by query types. • submit_final_sql: submit the final answer. The tool set and their implementations remain fixed across all skills and all experiments. Skills influence only the strategy text in the system prompt; they cannot add, remove, or modify tools. B.2
Practices for Better Skill Learning
We describe five practical lessons that proved important for making reflective skill optimization work reliably on Text-to-SQL agents. Proxy models for optimization. Running the full GEPA reflect-mutate-evaluate loop on the strongest available model (e.g., Opus 4.6) is both expensive and, perhaps counter-intuitively, less effective. Stronger models already produce near-correct trajectories on many training examples, leaving the reflector with subtle failure signals that are hard to attribute to specific strategy gaps. We find that using a weaker model from the same family as a proxy, e.g., Sonnet 4.6 for the Opus 4.6 experiments, yields faster and cheaper optimization while producing larger per-round accuracy gains that push skill evolution more aggressively. The resulting optimized skills transfer well to the stronger model: the strategy-level improvements (e.g., “anchor the grain before grouping”) are model-agnostic, even though they were discovered from the proxy’s more frequent and more interpretable failures. Brevity as a tiebreaker. During GEPA optimization, multiple candidate skill mutations often achieve the same accuracy gain on the hard batch. Rather than selecting arbitrarily, we break ties by preferring the shorter prompt. This 16
Residual Skill Optimization for Text-to-SQL Ensembles
A P REPRINT
acts as a lightweight regularizer: longer prompts tend to accumulate instance-specific details from the training batch, increasing the risk of overfitting to particular schemas or question patterns. Dialect- and instance-agnostic reflection. We explicitly instruct the reflector model to avoid including any instancespecific or dialect-specific details in the optimized strategy text, including concrete table names, column names, dialect-specific function syntax, or schema patterns observed in the training batch. The reflector is told that the skill must generalize across unseen databases, SQL dialects, and task formats. This constraint is important because the skills optimized on training data with Snowflake grammar are later evaluated on SQLite and BigQuery, making dialect-specific advice actively harmful. Correctness-only reward signal. We deliberately use binary execution correctness as the sole reward signal for GEPA optimization. Although finer-grained intermediate rewards—such as keyword overlap with gold SQL, structural similarity scores, or partial-credit metrics based on clause matching—might seem more informative, we found them prone to reward hacking in practice. For example, a keyword-coverage reward incentivizes skills that instruct the agent to speculatively include as many SQL keywords as possible, inflating the reward without improving correctness. Similarly, structural similarity rewards can penalize valid alternative query plans that differ from the gold SQL in form but not in semantics. Binary correctness avoids these pathologies: a skill is rewarded only if the agent’s final SQL produces the correct result on the target database, providing an unambiguous and ungameable training signal. Skill-order rotation. In Algorithm 1, each skill removes its solved examples before the next skill is optimized, so earlier skills in a batch see an easier and broader residual, while later skills are trained on a narrower and harder subset. To guarantee that each skill occasionally sees the full batch before other skills remove solved examples, we rotate the skill order across batches. This reduces positional bias and prevents later skills from being systematically specialized only to the hardest tail of failures. Note that if the number of batches is fewer than the number of skills, we apply a larger stride to the rotation to maintain uniform positional coverage.
17
Residual Skill Optimization for Text-to-SQL Ensembles
C
A P REPRINT
Skill Pool
We list the K = 8 seed skills used in our experiments, showing both the initial hand-designed seed prompt and the prompt after residual optimization on Snowflake training data. Each prompt is injected verbatim into the agent’s system message; the agent’s tool set and control flow remain unchanged. Optimized prompts are labeled with a round suffix (e.g., _r1) indicating the GEPA batch that produced the accepted mutation. default.
Strategy: the baseline agentic behavior—balanced exploration followed by incremental SQL construction.
Seed prompt:
default
## Strategy 1. EXPLORE first: run queries to understand the data–-check table structures, column values, data types, join keys, actual string values in the data 2. If unsure about any SQL function’s syntax or behavior, call lookup_docs BEFORE writing the query 3. For common patterns (top-N, running totals, pivots), call get_sql_pattern for a template 4. PLAN your approach based on what you discovered 5. WRITE and TEST your SQL incrementally–-run it via execute_sql to check results 6. VERIFY results look reasonable (right number of rows, right columns, sensible values) 7. Call review_sql to get a second opinion before submitting 8. SUBMIT only when confident Optimized prompt:
default
1. EXPLORE the data first (as in seed). 2. CLARIFY ambiguities–-identify potential traps: NULLs in key columns, case sensitivity, duplicate rows, date formats, and whether counts should be DISTINCT. 3. MAP the question to SQL primitives–-explicitly decide join type (INNER vs LEFT), filter placement (WHERE vs HAVING), aggregation scope, and NULL handling before coding. 4. Check templates–-call get_sql_pattern and lookup_docs (as in seed). 5. BUILD incrementally–-write and execute_sql each CTE or subquery alone (as in seed). 6. VALIDATE against the question–-re-read the question, then check: correct columns returned? correct filter conditions? DISTINCT where needed? NULL-safe denominators? ordering and limits match? 7. CROSS-CHECK edge cases–-run a quick sanity query (e.g., total counts, min/max values, a spot-check join) to confirm the final result is not inflated by fanout or deflated by over-filtering. 8. REVIEW–-call review_sql and address any flagged issues. 9. SUBMIT only after incremental checks and review pass.
direct_coder.
Strategy: drafts SQL immediately, refines through execution feedback.
Seed prompt:
direct_coder
## Strategy: DIRECT CODING You are an EFFICIENT SQL writer. 1. 2. 3. 4. 5. 6. 7.
Write SQL quickly, test, iterate.
Read the question carefully. Identify the core tables, joins, and aggregations needed. Write your best SQL attempt IMMEDIATELY based on the schema. Execute it. If errors occur, read the error message carefully and fix. If the query runs but results look wrong, investigate specific columns/values. Iterate rapidly–-each revision should fix one specific issue. Do NOT over-explore. Only investigate columns/values that are directly relevant to errors. SUBMIT as soon as the query produces reasonable results.
GEPA added lookup-table awareness and structured error-repair guidance. Key additions over the seed (new or substantially expanded material in bold): Optimized prompt: ## Strategy:
direct_coder
DIRECT CODING
1. Read the schema first. Before writing any SQL, identify ALL tables mentioned or implied by the question. Pay special attention to lookup/reference/static tables (e.g., category tables, node tables, type tables) that provide human-readable names or filter criteria–-these almost always require a JOIN.
18
Residual Skill Optimization for Text-to-SQL Ensembles
A P REPRINT
2. Map question terms to schema columns. If the question references a name, label, or category, find which table owns that column. Never filter or select on a column that doesn’t exist in the target table–-use the correct table via JOIN instead. 3. Write your best SQL IMMEDIATELY based on the schema. Use explicit JOIN conditions. When lookup tables exist, join them rather than filtering on raw IDs or payload strings. 4. Execute. Fix errors specifically: - Column-not-found → find the correct table and JOIN it - Wrong results → verify JOIN keys and WHERE filters match actual values - Missing rows → check JOIN type (INNER vs LEFT) 5. Aggregation and output checks: Verify GROUP BY includes all non-aggregated SELECT columns; verify ORDER BY, LIMIT, and NULLS LAST where appropriate; confirm output column names match the question. 6. Iterate rapidly–-each revision fixes one specific issue. 7. SUBMIT as soon as the query produces reasonable results. Key reminder: Questions involving categories, types, nodes, or classifications almost always require joining to a static/lookup table. Never assume the needed label lives in the fact table–-check the schema.
decompose.
Strategy: identifies sub-questions before composing the final SQL.
Seed prompt:
decompose
## Strategy: DECOMPOSE & CONQUER Break complex questions into simple subqueries, build bottom-up. 1. PARSE the question into atomic requirements: What is being counted/summed/averaged? What are the filter conditions? What are the grouping columns? Is there ranking, ordering, or limiting? 2. BUILD each piece as a standalone CTE: Start with the base data, add joins one at a time verifying row counts, add aggregations verifying results at each step. 3. COMPOSE CTEs into the final query using WITH...SELECT. 4. Run each CTE individually via execute_sql to verify intermediate results. 5. Call review_sql on the assembled final query before submitting. 6. SUBMIT only when confident.
GEPA expanded this into a structured six-step process with explicit grain-anchoring and filter-validation phases. Key additions (in bold): Optimized prompt: ## Strategy:
decompose
DECOMPOSE & CONQUER
Step 1: PARSE the question into atomic requirements–-explicitly identify: output columns/metrics, grain (one row per what?), filter conditions, grouping dimensions, and whether there is ranking, ordering, limiting, or a ratio/composition calculation. Step 2: ANCHOR the grain before grouping. Match GROUP BY columns precisely to the output grain–-no more, no less. If the question asks for monthly totals, group by month only; do not add route, city, or other columns unless explicitly requested. Step 3: BUILD each piece as a standalone CTE. For ratio/composition queries, compute totals in one CTE, subgroup counts in another, then JOIN and divide. Step 4: VERIFY join logic and filter semantics. Confirm join keys actually link the intended entities; avoid fan-out. Confirm filter values match the domain as they appear in the data, not as paraphrased in the question. Step 5: ASSEMBLE and REVIEW. Call review_sql; confirm output columns and grain match the question. Step 6: SUBMIT only when confident.
explore_heavy. Seed prompt:
Strategy: spends extra steps on schema and sample-row inspection before drafting. explore_heavy
## Strategy: DEEP EXPLORATION You are a THOROUGH explorer. Before writing ANY SQL, deeply understand the data. 1. CATALOG SCAN: List all schemas and tables. Check which tables have data (SELECT COUNT(*)). 2. COLUMN AUDIT: For each relevant table, run DESCRIBE or SHOW COLUMNS. Check actual column types.
19
Residual Skill Optimization for Text-to-SQL Ensembles
A P REPRINT
3. VALUE PROFILING: For key columns, run SELECT DISTINCT to see actual values, formats, ranges. 4. JOIN DISCOVERY: Test joins between tables with small queries before using them in final SQL. 5. Only after thorough exploration, write your query. 6. Test edge cases: What if there are NULLs? What if the join produces duplicates? 7. Call review_sql before submitting. 8. SUBMIT only when confident.
GEPA reorganized the strategy into six explicit phases and added an output-requirements clarification phase (Phase 4) that was absent from the seed. Key additions (in bold): Optimized prompt: ## Strategy:
explore_heavy
DEEP EXPLORATION
Phase 1–3: Schema Discovery, Value Profiling, Join Validation (as in seed). Phase 4: Clarify Output Requirements Before Writing SQL. Before coding, explicitly answer: - Granularity: one summary row, or one row per dimension? - Aggregation: which columns require AVG, SUM, COUNT? - Filters: date ranges, status filters, or categorical constraints implied? - Extra columns: would adding dimension columns change the granularity? If the question asks for summary metrics without specifying a grouping dimension, produce a single aggregated row–-do NOT return raw row-level data or add unrequested dimension columns. Phase 5: Write and Validate SQL–-apply NULLIF in denominators to avoid divide-by-zero; use CTEs to separate filtering, joining, and aggregation stages. Phase 6: Submit only when the output granularity, aggregation, and columns exactly match what was requested.
conservative.
Strategy: prefers the simplest faithful query; avoids speculative constructs.
Seed prompt:
conservative
## Strategy: CONSERVATIVE & SAFE Prefer simple, safe SQL. Avoid unnecessary complexity. 1. Start with the SIMPLEST possible query that could answer the question. 2. Avoid: Complex nested subqueries, window functions unless required, multiple joins when one will do, HAVING when WHERE suffices, correlated subqueries. 3. Do NOT add any clause the question didn’t ask for: No ROUND unless asked, no COALESCE unless NULLs are a proven problem, no extra WHERE filters or output columns. 4. When ambiguous, pick the LITERAL interpretation. 5. Test your query. If it works and looks reasonable, submit. 6. SUBMIT once results make sense. Don’t over-iterate.
GEPA added explicit guidance on AND-vs-OR filter logic and output granularity matching—two recurring failure modes in the training data. Key additions (in bold): Optimized prompt:
conservative
(Steps 1–3 retained from seed.) 4. Filter logic–-AND vs OR: - AND: both conditions must hold simultaneously on the same row. - OR: either condition suffices. - When filtering across related entities (e.g., origin OR destination), default to OR unless the question explicitly requires all conditions on the same record. 5. Output structure–-match the question’s requested granularity exactly: - Identify the grouping dimensions the question asks for before writing GROUP BY. - Do not substitute finer-grained groupings (e.g., day) when coarser ones (e.g., month) are requested. - Column names and aliases should reflect the question’s terminology. 6. When ambiguous, pick the LITERAL interpretation. Consult reference templates for grouping, join, and conditional aggregation patterns. 7. Test and submit once results make sense.
adversarial_checker.
Strategy: actively stress-tests the query for edge cases before finalizing.
20
Residual Skill Optimization for Text-to-SQL Ensembles
Seed prompt:
A P REPRINT
adversarial_checker
## Strategy: ADVERSARIAL SELF-CHECK After writing SQL, actively try to BREAK it before submitting. 1. Explore and write your initial SQL query. 2. Execute it and get results. 3. NOW, CHALLENGE your own query: Does the question ask for X but you computed Y? Could a join be producing duplicates? Are you filtering correctly? For ratios: verify numerator and denominator independently. For ‘top N’: verify ordering. 4. Fix any issues you discover. 5. Call review_sql for an independent check. 6. SUBMIT only after surviving both your own and the reviewer’s scrutiny.
GEPA restructured the strategy into four explicit phases, adding an upfront decomposition step and specific guidance for period-over-period comparisons and join-type selection: Optimized prompt:
adversarial_checker
Phase 1–-Decompose the Question Before Writing: Identify ALL computations required. If the question involves change/comparison, plan for two separate aggregations and a join or pivot–-never a single flat filter. If it involves rates or ratios, plan numerator and denominator explicitly. Sketch the output shape. Phase 2–-Write SQL Matching Full Complexity: Use CTEs for multi-step logic. Use window functions for rankings. Use FULL OUTER JOIN when comparing two periods where either side may have no data. Avoid the trap of “just filter and return raw rows” when computation is required. Phase 3–-Adversarial Challenge (expanded from seed): complexity check, output check, aggregation check, filter check, join type check (OUTER vs INNER), ratio/rate check. Phase 4–-Fix and Validate. Consult templates for period-over-period CTE patterns.
template_first.
Strategy: anchors query shape from retrieved SQL patterns before coding.
Seed prompt:
template_first
## Strategy: TEMPLATE-FIRST PLANNER Before writing SQL, call get_sql_templates and get_sql_pattern to anchor the query shape. Pick the closest template, adapt only the table names, columns, filters, grain, and ordering confirmed from the schema, then execute one focused validation query. Prefer template-guided CTE/window/ratio patterns over ad hoc exploration. Submit after the result shape matches the question.
GEPA appended a mandatory pre-submit checklist targeting schema hallucination—a failure mode where the agent references tables or columns it has not verified. Key addition (in bold): Optimized prompt:
template_first
(Seed text retained in full.) MANDATORY pre-submit checklist (do NOT skip): 1. Tables exist as named–-every table in your final SQL was confirmed to exist via execute_sql. 2. Columns exist as named–-every column appeared in actual schema output. 3. Identifier quoting is correct for the dialect. 4. Final query has executed at least once and returned non-empty rows whose shape matches the question. Submitting SQL that references non-existent columns, or that has never been successfully executed, is an automatic failure.
fast_error_repair. Seed prompt:
Strategy: prioritizes execution-error recovery over upfront planning. fast_error_repair
## Strategy: FAST ERROR REPAIR Move quickly: inspect only the most relevant tables, write the first plausible SQL early, execute it, and repair from concrete errors or wrong-shaped results. Do not exhaustively profile. Each iteration changes one thing: missing column, join key, filter value, aggregation grain, or dialect syntax. Submit as soon as the SQL runs and the output shape answers the question.
21
Residual Skill Optimization for Text-to-SQL Ensembles
A P REPRINT
GEPA expanded the single-paragraph seed into a five-phase process, adding a structured triage table and a value-sanitycheck phase. Key additions (in bold): Optimized prompt: Core loop:
fast_error_repair
write → run → verify values → repair → submit.
Phase 1–-Minimal Table Scan (as in seed). Phase 2–-First SQL Draft. Prefer CTEs for multi-step logic. Commit one join key and one filter assumption up front. Phase 3–-Execute & Triage. Change exactly one thing per iteration: Column not found → alias or rename Join returns 0 rows → relax join key or flip direction Count inflated → add DISTINCT or check fan-out join Wrong grain → re-examine GROUP BY columns Phase 4–-Value Sanity Check (the key addition): Before submitting: do not only check shape–-check values. Are counts suspiciously 0 or implausibly large? Does a percentage exceed 1.0 or go negative? Phase 5–-Submit when both shape and values are plausible.
22
Residual Skill Optimization for Text-to-SQL Ensembles
D
Additional Experimental Results
D.1
Variance and Candidate-Level Breakdowns
A P REPRINT
Table 7 reports the full Spider2-Lite Opus-4.6 table with standard deviations. The qualitative conclusion from the main text is unchanged after adding variance: D IV S KILL-SQL remains the strongest method on selected accuracy in all three dialects, with the largest gains on the more complex Snowflake and BigQuery settings. SQLite Method DIN-SQL R E F O RCE CHASE-SQL D IV S KILL-SQL
pass@1
pass@8
Snowflake Sel. acc.
pass@1
pass@8
BigQuery Sel. acc.
40.74±2.72 / / 0.97±0.40 55.28±2.62 / 57.78±2.72 43.47±2.09 / 50.72±2.25 62.59±2.47 76.30±2.59 63.70±2.59 51.21±1.94 68.60±2.01 53.14±2.33 62.13±2.04 73.33±1.82 64.44±1.62 60.08±1.59 72.46±1.55 64.25±1.52
pass@1
pass@8
Sel. acc.
18.54±1.71 51.10±2.23 / 55.12±2.28 59.02±2.00 71.71±2.03 56.59±2.28 62.07±1.84 73.17±1.67 64.88±1.12
Table 7: Spider2-Lite results with Opus 4.6. Why R E F O RCE selection can underperform mean candidate quality. In the main experiment, we noted that ReFoRCE’s final selected SQL can be worse than the average quality of its individual candidate runs. Table 8 makes this concrete. On SQLite, the mean per-candidate Pass@1 is 64.44 while the final selected accuracy is only 58.52; on Snowflake, the same gap is 47.83 vs. 41.06. This indicates that R E F O RCE often produces correlated candidates that agree on the same wrong answer, so majority voting can amplify a dominant failure mode rather than recover the strongest candidate. BigQuery is less pathological: selected accuracy (51.22) slightly exceeds the mean candidate accuracy (48.19), but still remains well below the oracle upper bound of 56.59. Here, “Max” is exactly Pass@4 because R E F O RCE exposes four candidate runs in this evaluation. ReFoRCE’s --num_votes parameter only sets the number of self-refinement threads per instance, not the number of surviving candidates: each thread is a 5-step refine loop that may terminate without writing any SQL (max-iter exhaustion, empty-result early-stop, invalid response, or oversized schema). In our GPT-5.4 run the realized candidate count per instance ranges over all threads, so a Pass@k oracle is ill-defined and not directly comparable to the Pass@8 results we report for the other methods. Pass@1 is defined as a complete set of first generated candidates for each question to show the interior intermediate results of ReFoRCE before majority voting. In this table, we simply give four sets of candidates to explain why ReFoRCE gets lower accuracy on Sel. acc. compared to so-called Pass@1. Dialect
cand 0 cand 1 cand 2 cand 3 Min
SQLite (n=135) 63.70 Snowflake (n=207) 41.55 BigQuery (n=205) 48.29
61.48 45.89 46.83
65.93 43.96 45.37
Max Mean Sel. acc.
65.93 53.33 74.81 64.44 44.44 38.65 56.52 47.83 46.34 41.95 56.59 48.19
58.52 41.06 51.22
Table 8: R E F O RCE per-candidate Pass@1 on Spider2-Lite for GPT-5.4
23
Residual Skill Optimization for Text-to-SQL Ensembles
D.2
A P REPRINT
Additional CHASE-SQL Comparison with GPT-5.4
Candidate-pool quality and rankability. The main Spider2-Lite results suggest that D IV S KILL-SQL’s advantage over CHASE-SQL is not only that it contains slightly more correct candidates, but also that its candidate pool is easier to rank. Tables 9 to 11 provide supporting evidence on the GPT-5.4 Spider2-Lite evaluation. Compared with CHASE-SQL, D IV S KILL-SQL improves pass@1 (56.95 vs. 54.96), pass@8 (78.79 vs. 76.42), and selected accuracy (64.90 vs. 60.69), while reducing the oracle–selector gap (13.89 vs. 15.72). The same-instance head-to-head analysis shows 47 D IV S KILL-SQL-only wins versus 24 CHASE-SQL-only wins on final selection, and McNemar’s test confirms that the selected-accuracy gain is statistically significant (p = 0.0086). The candidate-density view further shows that D IV S KILL-SQL yields fewer dead pools with zero correct candidates, slightly more rich pools with 6–8 correct candidates, and fewer exact duplicate slots. Together, these results support the claim that residual skill optimization improves both candidate quality and candidate rankability, rather than merely increasing surface-form variation. Method
0 corr. 1 corr. 2–3 corr. 4–5 corr. 6–8 corr.
CHASE-SQL D IV S KILL-SQL
23.58 21.21
6.76 6.95
10.97 10.60
11.33 12.43
47.35 48.81
Table 9: Distribution of the number of correct candidates per instance in the 8-candidate pool. D IV S KILL-SQL pool yields fewer dead pools (0 correct), slightly more rich pools (6–8 correct).
Dialect
Ours-only sel.
CHASE-only sel.
Ours-only oracle
CHASE-only oracle
Local Snowflake BigQuery
12 16 19
5 8 11
6 11 12
5 6 5
Total
47
24
29
16
Table 10: Same-instance head-to-head on Spider2-Lite GPT-5.4. “Ours-only sel.” counts instances where the final selected SQL is correct for D IV S KILL-SQL but not for CHASE-SQL. “Ours-only oracle” counts instances where the 8-candidate pool contains at least one correct SQL for D IV S KILL-SQL but not for CHASE-SQL.
Comparison
N
D IV S KILL-SQL
CHASE-SQL
Ours-only wins
CHASE-only wins
Exact p
Final selected accuracy (all instances) Oracle pass@8 coverage (all instances) Final accuracy on jointly solvable instances
547 547 402
64.90 78.79 86.32
60.69 76.42 81.34
47 29 39
24 16 19
0.0086 0.0725 0.0119
Table 11: Paired same-instance comparisons on Spider2-Lite GPT-5.4 using the exact McNemar test. The strongest effect is on final selected accuracy, including the subset where both methods already contain at least one correct candidate somewhere in the pool. This supports the claim that the low-temperature D IV S KILL-SQL pool is easier to rank, not merely that it has slightly better oracle coverage.
24