arXiv CS open access

Are These Modules Worth Their Cost? A Paradigm-Level Accuracy-Cost Analysis of In-context Learning Text-to-SQL

2026 · arxiv_cs

arXiv CS · Papers · 2026

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

arXiv:2608.28432v1 [cs.CL] 28 Aug 2026

Are These Modules Worth Their Cost? A Paradigm-Level Accuracy-Cost Analysis of In-context Learning Text-to-SQL Jiayan Lin∗

Yujia Liu∗

Zijin Hong∗

Jinan University Guangzhou, China ljiayan@stu2023.jnu.edu.cn

Jinan University Guangzhou, China yujialiu@stu2023.jnu.edu.cn

The Hong Kong Polytechnic University Kowloon, Hong Kong zijin.hong@connect.polyu.hk

Zheng Yuan

Yilin Xiao

Hao Chen

The Hong Kong Polytechnic University Kowloon, Hong Kong yzheng.yuan@connect.polyu.hk

The Hong Kong Polytechnic University Kowloon, Hong Kong yilin.xiao@connect.polyu.hk

City University of Macau Taipa, Macau sundaychenhao@gmail.com

Qinggang Zhang

Xiao Huang

Feiran Huang

Jilin University Changchun, China qinggangzhang@jlu.edu.cn

The Hong Kong Polytechnic University Kowloon, Hong Kong xiao.huang@polyu.edu.hk

Beihang University Beijing, China huangfr@buaa.edu.cn

Abstract

Keywords

Recent advances in in-context learning (ICL) text-to-SQL have substantially improved execution accuracy on public benchmarks by assembling increasingly elaborate pipelines around the base generator, yet existing studies typically report aggregate end-to-end accuracy, without quantifying the marginal accuracy-cost contribution of individual design choices. Consequently, providing a unified, paradigm-level cost-accuracy quantification remains a critical challenge for understanding and configuring modern textto-SQL. To address this, we instantiate 17 paradigm-level configurations across five recurring modules of the ICL text-to-SQL pipeline under a single controlled implementation, and attribute each paradigm’s marginal contribution and incurred cost across all four backbones spanning diverse capability levels and reasoning styles. Our analysis reveals that execution-feedback refinement is the only paradigm whose benefit holds universally at consistently low cost, while most other modules help only under backbone-dependent conditions. Token accounting shows that input demand is more closely tied to pipeline structure, whereas output demand is more sensitive to backbone generation behavior. Cross-module analysis further shows that stacking improves accuracy on most backbones, although how the gains compose varies with backbone capability. We also find that a fixed budget is often better spent engineering a more elaborate pipeline over a mid-tier backbone than upgrading to a frontier model with a lean pipeline. These findings distill into an actionable, cost-aware tiered guideline that transfers to five additional backbones without per-paradigm search.

Text-to-SQL, In-context Learning, Large Language Models, AccuracyCost Trade-offs

CCS Concepts • Information systems → Relational database query languages; • Computing methodologies → Natural language processing. ∗ All authors contributed equally to this research.

Introduction

The advent of large language models (LLMs) has driven remarkable progress on text-to-SQL [13], a foundational task that translates natural language questions into executable SQL queries over relational databases. In particular, in-context learning (ICL) approaches [34] have substantially pushed execution accuracy on widely used benchmarks such as BIRD [19] and Spider [41] by assembling increasingly elaborate pipelines around the base generator. ICL is particularly suitable for such analysis because its improvements are largely realized through inference-time pipeline design, allowing the marginal benefit and added cost of individual components to be measured while holding the underlying backbone fixed. A modern ICL textto-SQL system rarely issues a single prompt: it retrieves in-context demonstrations [29], links the relevant schema [18], samples and selects among multiple candidates [15], and refines the chosen query through execution feedback or multi-agent coordination [33]. Each successive system stacks more such modules atop the last, propelling state-of-the-art accuracy to new heights on public leaderboards. These increasingly modular pipelines still create a growing design space in which accuracy gains may often arise in practice from very different combinations of inference-time components. However, is the accuracy growth on text-to-SQL leaderboards reliably worth its disproportionately growing cost? Figure 1 illustrates a concerning trend: execution accuracy keeps rising as more modules are stacked, while the cost required to gain each additional point grows disproportionately. Pipeline components, however, are typically reported only through a single end-toend accuracy number for the system. This aggregate-only reporting practice raises two concerns: 1) The marginal contribution of

Jiayan Lin, Yujia Liu, Zijin Hong, Zheng Yuan, Yilin Xiao, Hao Chen, Qinggang Zhang, Xiao Huang, and Feiran Huang

What is the grade span offered in the school with the highest longitude? General ICL Text-to-SQL Pipeline: Pre-Processing

SQL Generation

Post-Processing

Baseline Incorrect SQL User Question Generating SQL... SELECT GSoffered FROM schools Schema ORDER BY longitude DESC LIMIT 1 Evidence: maximum absolute longitude Missing logic: ABS(longitude)

Outcome Low Accuracy Low Cost

Pipeline Stack Configuration Direct Generation

Masked Question

SQL Skeleton

0 extra calls

+1 LLM call

Agentic Linking

Keyword – Column Draft – Schema Propose – Verify Encoder only

+1 draft call

Identify Target — Interpret Evidence — Construct SQL

What to return? User Question

+� LLM calls

{SQL₁, SQL₂, …, SQL₅} SELECT GSoffered...DESC LIMIT 1

SELECT GSoffered ...DESC LIMIT 1

Refinement

Decomposition-Based

Schema Linking Pseudo-SQL Guided

SELECT GSoffered...DESC LIMIT 1

CoT-Enhanced

Question Similarity

RetrievalBased

Candidate Selection

Generation Strategy

Example Retrieval Retrieve demonstrations based on:

Refined SQL

Selected SQL

Outcome

GSoffered

Which school?

Max ABS(longitude)

How to select?

DESC + LIMIT 1

SELECT GSoffered ...

Higher Accuracy Higher Cost

Backbone–Pipeline Trade-offs Small Backbone

Mid-Tier Backbone

Frontier Backbone

Accuracy Accuracy Accuracy

Cost Cost Cost

Practical cost-aware guidance

Figure 1: Overview of a modular ICL Text-to-SQL pipeline and its accuracy–cost trade-offs. Five recurring modules span preprocessing, SQL generation, and post-processing, but their benefits and inference costs vary across backbones, yielding different cost-efficient configurations under fixed budgets. individual modules is overshadowed by aggregate accuracy, as the same end-to-end number may be reached through entirely different module combinations, leaving unclear which paradigms carry the gain and which are merely along for the ride; 2) Cost is rarely accounted for jointly with accuracy, despite agentic refinement loops and multi-candidate sampling consuming several times, and in some cases an order of magnitude, more tokens and dollars than a single-pass baseline. These concerns make it difficult for practitioners to decide which modules are worth adding for a backbone, and how far the pipeline should go before engineering ceases to pay off. They therefore motivate us to study this recurring class of modular ICL pipelines under controlled settings, where inference-time modules can be varied while holding the backbone fixed to more reliably attribute their marginal accuracy and cost. Studies have begun to interrogate aspects of text-to-SQL pipelines, such as ablations on schema linking [22] or comparisons among candidate selection strategies [24], while several surveys catalog representative methods and their design choices [17]. However, these efforts either evaluate isolated components within a single system or report each method as a monolith without disentangling its constituent paradigms, leaving open how each module’s contribution shifts across backbones of differing capability and reasoning style. In this context, effectively quantifying the marginal accuracy–cost tradeoff of each design paradigm in ICL text-to-SQL pipelines remains a significant challenge. As a solution, this paper conducts a controlled paradigm-level evaluation that decomposes the ICL text-to-SQL pipeline into recurring modules, instantiates each design choice as a paradigm-level configuration under a unified implementation, and isolates its marginal contribution alongside the incurred cost across backbones spanning

diverse capability levels and reasoning styles. Unlike accuracy-only evaluations, we jointly examine execution accuracy and inference cost, revealing how each paradigm’s value shifts with model capability. Overall, our contributions in this paper are listed as follows: • We conduct the first controlled, paradigm-level accuracy–cost evaluation of in-context learning (ICL) text-to-SQL pipelines, instantiating 17 paradigm-level configurations across five recurring modules under a single unified implementation. • We reveal that execution-feedback refinement is the only paradigm whose benefit holds universally at consistently low cost, while most other paradigms instead help only under backbone-dependent conditions, exposing the unreliability of aggregate-only accuracy reporting in practice. • We further show that a fixed budget is often better spent engineering a more elaborate pipeline over a mid-tier backbone than upgrading to a frontier model with a lean pipeline, and distill these findings into a tiered cost-efficient configuration guideline validated on five additional backbones across capability levels without per-paradigm search.

Pipeline Decomposition

We decompose the ICL Text-to-SQL pipeline into three main stages, Pre-Processing, SQL Generation, and Post-Processing, comprising five recurring functional modules: Example Retrieval, Schema Linking, Generation Strategy, Candidate Selection, and Refinement. These yield 17 paradigm-level configurations; Figure 2 summarizes the taxonomy and 24 representative methods. For a question 𝑞𝑖 with schema S𝑖 , the five modules are formalized below. (1) Example Retrieval. Retrieves a demonstration set Z𝑖 from Dtrain using a paradigm-specific representation 𝜌𝑒 : Z𝑖 = TopK sim 𝜌𝑒 (𝑞𝑖 ), 𝜌𝑒 (𝑞𝑧 ) . (1) 𝑧 ∈ Dtrain

Here, 𝑞𝑧 denotes the question in example 𝑧. 𝐸 1 Question Similarity directly uses raw question embeddings [17]; 𝐸 2 Masked Question Similarity masks schema-specific entities before retrieval [8]; and 𝐸 3 SQL Skeleton Similarity uses structural SQL query forms [27]. (2) Schema Linking. For each schema element 𝑢 ∈ S𝑖 , a paradigmspecific rule ℓ𝑠 (𝑢 | 𝑞𝑖 , S𝑖 ) ∈ {0, 1} determines whether 𝑢 is retained: e𝑖 = {𝑢 ∈ S𝑖 | ℓ𝑠 (𝑢 | 𝑞𝑖 , S𝑖 ) = 1} ⊆ S𝑖 . S

(2)

𝑆 1 Retrieval-Based Linking scores relevant schema elements via dense retrieval [38]; 𝑆 2 Pseudo-SQL Guided Linking retains elements explicitly referenced by a preliminary SQL draft [29]; 𝑆 3 Direct LLM Linking uses 𝑆 3𝑎 Single-shot [25] or 𝑆 3𝑏 Two-stage [6] LLM linking; and 𝑆 4 Agentic Linking uses iterative proposer-verifier exchanges. (3) Generation Strategy. Generates a pool Y𝑖 of 𝑁 SQL candidates: n o𝑁 e𝑖 , Z𝑖 ; 𝑁 . Y𝑖 = 𝑦ˆ𝑖(𝑘 ) = Generate𝑔 𝑞𝑖 , S (3) 𝑘=1

e𝑖 = S𝑖 . Disabled retrieval uses Z𝑖 = ∅, while disabled linking uses S 𝐺 1 Direct Generation generates SQL directly [4]; 𝐺 2 CoT-Enhanced Generation elicits intermediate reasoning [24]; and 𝐺 3 DecompositionBased Generation decomposes the question before generating the final SQL [34]. They may also be combined via parallel generators.

Are These Modules Worth Their Cost? A Paradigm-Level Accuracy-Cost Analysis of In-context Learning Text-to-SQL 2.Masked Question Similarity DAIL-SQL DEA-SQL

N-rep

MCS-SQL

DeepEye-SQL

OpenSearch-SQL

XiYan-SQL

PET-SQL

Alpha-SQL

3.SQL Skeleton Similarity

4.Agentic Linking

Gen-SQL

PET-SQL

Single-shot

Two-stage CHASE-SQL

ReFoRCE DeepEye-SQL

MCS-SQL DEA-SQL

LinkAlign

N-rep

SuperSQL

AutoLink

Alpha-SQL RSL-SQL

DEA-SQL MCS-SQL

MAC-SQL

DIN-SQL

RSL-SQL

OpenSearch-SQL

XiYan-SQL

DeepEye-SQL

Gen-SQL

3.Direct LLM Linking

Gen-SQL

PURPLE

SuperSQL

DeepEye-SQL

SuperSQL

OpenSearch-SQL

PURPLE

1.Question Similarity DAIL-SQL

2.Pseudo-SQL Guided Linking

1.Retrieval-based Linking

Agentar-Scale-SQL

CHESS-SQL

XiYan-SQL

C3-SQL

ACT-SQL

1.Execution-Feedback Refinement Gen-SQL

RSL-SQL

XiYan-SQL

AutoLink

1. Direct Generation

ICL Text-to-SQL Pipeline

CHESS-SQL Agentar-Scale-SQL

Alpha-SQL

Agentar-Scale-SQL

MAGIC Agentar-Scale-SQL

3.Agentic Refinement CHASE-SQL

1.Execution-Based Voting

ReFoRCE

SuperSQL DeepEye-SQL

MAC-SQL

PET-SQL OpenSearch-SQL

DeepEye-SQL

RSL-SQL DEA-SQL SuperSQL

PURPLE Gen-SQL

DAIL-SQL

PET-SQL

2. CoT-Enhanced Generation

2.Self-Correction

DIN-SQL

ReFoRCE

N-rep

DeepEye-SQL

OpenSearch-SQL

DEA-SQL

C3-SQL

XiYan-SQL

ReFoRCE

CHASE-SQL

C3-SQL

DAIL-SQL PURPLE

N-rep

2.LLM Preference Selection

Scoring and Ranking Pairwise Comparison

Agentar-Scale-SQL

OpenSearch-SQL

DIN-SQL

DeepEye-SQL

MCS-SQL MAC-SQL

Alpha-SQL

CHESS-SQL CHASE-SQL

ACT-SQL

AutoLink

3. Decomposition-Based Generation AutoLink

DeepEye-SQL

Alpha-SQL

CHASE-SQL

XiYan-SQL

AutoLink

RSL-SQL

CHESS-SQL

CHASE-SQL

N-rep

MCS-SQL

DeepEye-SQL

MAC-SQL

Agentar-Scale-SQL

AutoLink Alpha-SQL

RSL-SQL

DIN-SQL

Figure 2: Taxonomy of ICL Text-to-SQL pipelines, organizing five modules across three pipeline stages into 17 paradigm-level configurations and 24 representative methods, including methods that naturally span multiple paradigms within the pipeline. Repeated method names indicate systems that combine design paradigms across modules or use multiple strategies within the same module, providing a unified view of how modern ICL Text-to-SQL systems assemble pipeline components in practice. (4) Candidate Selection. Selects one query from Y𝑖 according to selection paradigm 𝑐: 𝑁 𝑦ˆ𝑖sel = Select𝑐 {𝑦ˆ𝑖(𝑘 ) }𝑘=1 ∈ Y𝑖 . (4)

finally detail the controlled implementation of the 17 paradigm configurations to ensure consistent and comparable assessment across paradigms and backbones within our unified framework.

𝐶 1 Execution-Based Voting selects from the largest result-equivalence group [20]; 𝐶 2 LLM-Based Preference Selection uses an LLM to choose via 𝐶 2𝑎 Scoring and Ranking [9] or 𝐶 2𝑏 Pairwise Comparison [16].

3.1

(5) Refinement. Maps the selected SQL 𝑦ˆ𝑖sel to the final prediction under refinement paradigm 𝑟 : e𝑖 . 𝑦ˆ𝑖 = Refine𝑟 𝑦ˆ𝑖sel, 𝑞𝑖 , S (5)

Dataset and Backbones

Dataset. We conduct the main paradigm-level analysis on the development split of BIRD [19], which contains 1,534 question and SQL pairs over 11 databases. BIRD is widely used in recent ICL Textto-SQL work and provides evidence annotations that are directly relevant to several paradigms studied in this paper. To examine cross-benchmark generalization, we additionally use Spider [41] (1,034 samples) to evaluate cross-benchmark transfer of the BIRDderived pipeline configurations under the same protocol in § 5.2.

𝑅1 Execution-Feedback Refinement rewrites only after an execution error or empty result [2]; 𝑅2 Self-Correction performs prompt-level review without execution [39]; and 𝑅3 Agentic Refinement coordinates multiple revision roles using execution feedback [33]. Without refinement, the selected SQL is simply returned unchanged [10, 26]. Our taxonomy covers 24 representative ICL Text-to-SQL methods published between 2023 and 2026 that report on BIRD or Spider and provide sufficient pipeline detail for reliable paradigm-level classification. Methods may span multiple paradigms, especially through parallel generation or combined schema linking signals.

Backbones. We run the full set of paradigm configurations on four primary backbones spanning different capability levels and reasoning behaviors: GPT-4o-mini, Gemini-2.5-Flash, DeepSeekV4-Flash (reasoning), and GPT-5.4. To further examine the effect of reasoning behavior on generation strategies, we additionally evaluate Qwen3-235B in thinking mode and o3-mini in § 4.2. Finally, we evaluate the tiered guideline in § 6 on five additional backbones by applying the three prescribed tiers without per-paradigm search.

3.2

Experimental Setup

This section describes the experimental setup for our paradigmlevel analysis. We first introduce the datasets and backbone scope, then present the evaluation metrics and statistical testing, and

Metrics and Statistical Testing

Accuracy metrics. For intuitive comparison, we follow the standard evaluation setting in the text-to-SQL community [41]. Our primary evaluation metric is execution accuracy (EX). Given a

Jiayan Lin, Yujia Liu, Zijin Hong, Zheng Yuan, Yilin Xiao, Hao Chen, Qinggang Zhang, Xiao Huang, and Feiran Huang

set D = {(𝑞𝑖 , 𝑦𝑖 )}𝑛𝑖=1 of question and gold-SQL pairs and predicted SQLs {𝑦ˆ𝑖 }𝑛𝑖=1 : 𝑛

1 ∑︁ 1[ Exec(𝑦ˆ𝑖 ) = Exec(𝑦𝑖 ) ] , EX = 𝑛 𝑖=1

(6)

where Exec(·) denotes the execution operator on the corresponding database. For configurations that sample multiple candidates, 𝑁 denote the 𝑁 sampled candidates for example 𝑖. We let {𝑦ˆ𝑖(𝑘 ) }𝑘=1 additionally report oracle candidate recall (OCR): OCR =

𝑛 𝑁 i 1 ∑︁ Ü h 1 Exec(𝑦ˆ𝑖(𝑘 ) ) = Exec(𝑦𝑖 ) , 𝑛 𝑖=1

(7)

𝑘=1

which upper-bounds the EX achievable by any selection strategy over a fixed candidate pool. Cost and resource metrics. To compare paradigms along the accuracy–cost tradeoff, for a paradigm 𝑝 instantiated in module 𝑚, we report cost per EX point (CPP): CPP(𝑝) =

Cost(𝑝) − Cost(𝑏𝑚 ) , 100 [EX(𝑝) − EX(𝑏𝑚 )]

(8)

where 𝑏𝑚 denotes the matched reference for module 𝑚, using the same sampling budget as 𝑝 (𝑁 =5 for Candidate Selection and 𝑁 =1 otherwise). Measured in USD/pp, CPP represents the additional cost to gain one percentage point of EX over the matched reference. CPP is reported only when 𝑝 strictly improves EX at strictly higher cost; paradigms that Pareto-dominate the reference are marked separately. We report all costs in USD and also record mean perquestion input (IN) and output (OUT) token counts for comparison to better characterize resource overhead beyond aggregate API cost.

Controlled Paradigm Implementation Require: 𝑞𝑖 , S𝑖 , optional evidence, backbone, target module 𝑚 , and paradigm setting Ensure: final prediction 𝑦ˆ𝑖 1: Keep all non-target modules at their trivial settings. 2: if 𝑚 is Example Retrieval then 3: Fix 𝑘=3, the training retrieval pool, and the same frozen encoder.  𝐸1,   retrieve with 𝑞𝑖 ,  4: Z𝑖 ← mask schema entities → retrieve, 𝐸 2 ,   draft SQL → skeleton → retrieve, 𝐸 . 3  5: else if 𝑚 is Schema Linking then 6: Fix the full-schema input and always retain primary/foreign keys.  retrieval-based filtering, 𝑆1,     draft SQL → referenced schema, 𝑆 2 ,    e𝑖 ← one-shot table/column selection, 𝑆 3𝑎 , 7: S    tables → columns, 𝑆 3𝑏 ,    𝑆4 .  propose → verify → [ revise ], 8: else if 𝑚 is Generation Strategy then 9: Fix one backbone generation call at temperature 0.  𝐺1,  generate SQL directly,   10: Y𝑖 ← reason → generate final SQL, 𝐺2,   decompose → generate final SQL, 𝐺 . 3  11: else if 𝑚 is Candidate Selection then 12: Fix the same 𝑁 =5 pool Y𝑖 sampled from 𝐺 1 . 13: Execute all candidates in Y𝑖 .  𝐶1,   result-equivalence voting,  14: 𝑦ˆ𝑖sel ← score/rank with execution outcomes, 𝐶 2𝑎 ,   pairwise tournament, 𝐶 2𝑏 .  15: else if 𝑚 is Refinement then 16: Fix the same inputs and temperature 0; keep the last valid SQL on failure.  𝑅1 ,   execute → [ rewrite if error/empty ],  17: 𝑦ˆ𝑖 ← review without execution feedback, 𝑅2 ,   correct with execution feedback → verify → [ correct ], 𝑅 . 3  18: end if 19: Run the remaining pipeline stages at their trivial settings. 20: return 𝑦ˆ𝑖 Gray italics: shared fixed settings.

Statistical testing. All EX gains over the matched reference are evaluated using McNemar’s test. We report 95% confidence intervals and determine statistical significance at 𝛼 = 0.05; detailed numerical intervals are provided in the supplementary material for reference.

3.3

Controlled Implementation

To ensure that quantitative comparisons reflect paradigm-level differences rather than incidental properties of specific systems [13], we re-implement the 17 configurations identified in § 2 as paradigm representatives under a unified controlled framework. Rather than reproducing any particular system, each configuration instantiates the defining mechanism of one paradigm while keeping shared implementation settings fixed within corresponding module across all backbones. Figure 3 summarizes these controlled implementations across all five modules. Detailed implementation settings and fallback behavior are provided in the final supplementary material. Baseline and single-swap protocol. We define a global baseline for each backbone with no retrieved demonstrations, the full schema, direct generation (𝐺 1 ) with 𝑁 =1, and neither Candidate Selection nor Refinement. For each target module, we replace only that module with the paradigm under study while keeping other modules at their trivial settings, so that the observed difference can then be attributed directly to the target paradigm. Candidate Selection is the only exception to the 𝑁 =1 reference: because selection requires multiple candidates, we use a matched 𝑁 =5 baseline that returns the first candidate from the same candidate pool, thereby

[·] denotes a conditional operation.

Figure 3: Controlled implementation of the 17 paradigm configurations. For each target module, shared settings are fixed and all non-target modules remain at their trivial settings, so that only the paradigm-specific mechanism itself varies. also separating the selector’s contribution from the gain due to multi-sample generation. For Generation Strategy, we additionally evaluate 𝑁 =5 sampling to examine effect on candidate-pool quality.

Single-Module Analysis

We instantiate each of the 17 paradigm configurations by replacing exactly one module in the global baseline pipeline, holding the remaining four at their trivial settings. This single-swap protocol isolates the marginal contribution of each paradigm. Table 1 reports the accuracy–cost and token statistics across all four backbones, while Figure 4 summarizes the ΔEX results with 95% confidence intervals. We organize the analysis around two questions below: Q1. Are there paradigms whose marginal benefit is robust across all four backbones, making them safe pipeline defaults? Q2. For paradigms whose benefit is not universal, does the variation correlate with model capability or reasoning style?

4.1

Universal Paradigms

Across the 17 overall paradigm configurations, 𝑅1 (ExecutionFeedback Refinement) is the only paradigm whose benefit holds

Are These Modules Worth Their Cost? A Paradigm-Level Accuracy-Cost Analysis of In-context Learning Text-to-SQL

Table 1: Module-level accuracy–cost results on BIRD dev across four LLMs. EX: execution accuracy (%); Cost: total API spend (USD); CPP: incremental USD per EX point over the matched-budget baseline (𝑁 =5 for Candidate Selection, 𝑁 =1 otherwise; § 3.2). IN/OUT: mean per-question tokens (DeepSeek-V4-Flash reasoning tokens count as OUT). Significance uses McNemar’s test (𝛼=0.05; full CIs in the supplement). In CPP, numbers denote significant gains; “◦ ”, nonsignificant gains; “–”, no EX improvement; and “∗ ”, a significant gain at no higher cost. / mark the best/second-best results; OCR@5 is the oracle upper bound for 𝑁 =5. GPT-4o-mini

Paradigm

EX↑ Cost↓ CPP↓

IN↓

DeepSeek-V4-Flash OUT↓ EX↑ Cost↓ CPP↓

Gemini-2.5-Flash

IN↓

OUT↓ EX↑ Cost↓ CPP↓

IN↓

GPT-5.4 OUT↓ EX↑ Cost↓ CPP↓

IN↓

OUT↓

Baselines

𝑁 =1 𝑁 =5 OCR@5

49.74 49.67 56.91 Example Retrieval

0.394 1.972 1.972

– – –

50.07 50.65 49.80

0.462 0.525 0.867

0.144

49.87 52.15 51.11 51.83 52.93

0.275 0.574 0.544 0.582 1.260

49.74 53.98 51.24 Candidate Selection

𝐸1 𝐸2 𝐸3

1510 7550 7550

51 255 255

56.32 56.45 60.56

0.547 2.756 2.756

– – –

1557 7787 7787

494 2523 2523

59.84 60.04 64.54

1.035 5.219 5.219

– – –

1642 8222 8222

73 374 374

61.28 61.41 66.82

7.305 35.864 35.864

– – –

1509 7545 7545

66 301 301

1806 2009 3357

50 68 103

56.06 56.84 57.11

0.613 0.717 1.063

– 0.326 0.660

1858 2311 3708

498 512 622

60.43 61.28 60.10

1.270 1.632 2.421

0.401 0.417 ◦

1842 2207 3358

110 161 228

62.58 63.69 63.49

8.602 9.613 14.879

0.995 0.957 3.417

1834 2058 3325

68 75 92

0.075 0.109 0.090 0.271

873 1844 1936 1528 4478

81 163 107 250 249

55.93 57.04 56.19 55.87 56.52

0.360 0.899 0.782 0.757 1.380

– 0.491 – – ◦

964 1871 1852 1634 4520

357 1157 896 944 953

59.13 61.15 60.89 60.76 61.34

1.117 1.784 1.659 1.808 2.870

– 0.575 0.599 0.847 1.224

857 1855 1924 1577 4524

188 243 202 282 205

60.63 62.39 60.82 59.97 62.06

5.102 10.955 9.672 12.806 23.631

– 3.293 – – 20.870

826 1903 1944 1622 4568

84 159 96 286 266

0.394 0.503 0.534

– 0.026 0.093

1510 1534 1574

51 163 187

56.32 55.93 56.13

0.547 0.556 0.651

– – –

1557 1602 1650

494 493 689

59.84 62.71 60.76

1.035 1.522 1.925

– 0.170 0.976

1642 1538 1590

73 212 311

61.28 62.84 63.10

7.305 10.000 11.476

– 1.723 2.285

1509 1533 1573

66 179 237

51.11 50.52 51.17

1.972 2.625 4.056

0.771 1.389

7550 9718 14962

255 423 666

57.30 59.00 58.54

2.756 3.602 4.743

0.333 0.953

7787 10319 14962

2523 3227 3562

60.89 61.60 61.93

5.219 8.045 10.622

1.806 2.858

8222 10018 14846

374 896 988

62.71 63.82 63.89

35.864 53.348 75.436

7.249 15.975

7545 9964 14933

301 658 790

54.63 51.83 53.26

0.471 1.030 1.365

0.016 0.305 0.276

1750 3509 4980

74 242 238

58.15 56.58 59.97

0.620 1.253 1.878

1697 3638 5007

596 1098 1870

62.32 60.95 63.04

1.099 3.211 4.281

0.026 1.964 1.016

1671 3513 5116

86 416 502

63.30 62.06 64.02

7.905 21.918 28.578

0.297 18.680 7.769

1592 3518 5251

78 366 367

◦ ◦

Schema Linking

𝑆1 𝑆2 𝑆 3𝑎 𝑆 3𝑏 𝑆4

◦

SQL Generation

𝐺1 𝐺2 𝐺3

𝐶1 𝐶 2𝑎 𝐶 2𝑏

∗

Refinement 0.040 ◦

0.365

across all four backbones at consistently low cost. It yields gains between +1.83 and +4.89 pp, with CPP below 0.30 USD/pp throughout. The mechanism is: a diagnostic execution is run on the candidate query, and the model is guided to rewrite only when execution errors or returns no rows. The call is therefore failure-gated rather than mandatory, which keeps the average overhead small while preserving the upside on the queries that actually need revision. The other two refinement paradigms illustrate why 𝑅1 ’s combination is rare. 𝑅2 (Self-Correction) significantly improves EX on three backbones but has no significant effect on the reasoning backbone DeepSeek-V4-Flash (+0.26 pp, n.s.); even where it helps, its CPP is far above 𝑅1 ’s. 𝑅3 (Agentic Refinement) is positive on all four backbones and posts the highest absolute EX in several columns, but at CPP one to two orders of magnitude above 𝑅1 . The pattern across the three paradigms is consistent: execution feedback supplies information the model did not have at generation time, while prompt-internal correction adds only what the model could in principle have produced on its own. Among universally helpful paradigms, this information asymmetry, rather than reasoning quality, is thus what separates cheap from expensive in this setting.

4.2

Model-Dependent Paradigms

The remaining four modules deliver gains that depend on the backbone. Rather than treating each module in isolation, we organize the discussion by backbone characteristics, since the modules’ contributions move together with model capability across tested backbones.

Schema Linking

Example Retrieval

SQL Generation

Candidate Selection

Refinement

6 GPT-4o-mini DeepSeek-V4-Flash

Gemini-2.5-Flash GPT-5.4

4 3

ΔEX (pp)

𝑅1 𝑅2 𝑅3

2 1 0

−1 −2 E1

S3a

S3b

C2a

C2b

Figure 4: ΔEX (pp) relative to the matched-budget reference baseline for 17 paradigm configurations on BIRD dev. Shaded bands show 95% CIs for paired differences, and the dashed line marks ΔEX = 0. Statistical significance is assessed using McNemar’s test. Full numerical CIs are provided in the supplementary material for every paradigm on each backbone. Capability gradient: schema linking and example retrieval exhibit opposing trends. On GPT-4o-mini, the weakest backbone in our set, Schema Linking delivers its largest gains across the four backbones: 𝑆 4 (Agentic Linking) adds +3.19 pp, and 𝑆 2 , 𝑆 3𝑎 , 𝑆 3𝑏 all clear +1.3 pp. The benefit shrinks sharply on stronger backbones. On GPT-5.4, only 𝑆 2 and 𝑆 4 yield positive gains, both at substantially higher CPP than pre-processing paradigms typically incur on cheaper backbones. Schema linking acts as a capability prosthesis: its value falls as the backbone’s schema-grounding ability

Jiayan Lin, Yujia Liu, Zijin Hong, Zheng Yuan, Yilin Xiao, Hao Chen, Qinggang Zhang, Xiao Huang, and Feiran Huang

Paradigm

EX↑

Qwen3-235B (Thinking)

𝐺1 𝐺2 𝐺3

57.82 58.02 (+0.20) 57.30 (−0.52)

4.55 5.22 5.84

𝐺1 𝐺2 𝐺3

54.11 53.46 (−0.65) 53.06 (−1.04)

5.67 5.83 7.08

o3-mini

Cost↓ CPP↓ IN↓ OUT↓ – 1555 3.426 1529 – 1596 – – –

1509 1546 1538

rises. Recent studies [22, 42] report that sufficiently strong models can generate correct SQL even without explicit schema linking; we extend the observation to a four-backbone, cost-aware setting. Example Retrieval shows the opposite trend, though weakly. 𝐸 2 (Masked Question Similarity) is the best retrieval paradigm on every non-reasoning backbone, and its gain scales with baseline EX, from +0.91 pp on GPT-4o-mini to +2.41 pp on GPT-5.4. One reading is that stronger non-reasoning backbones extract reusable structure from demonstrations, but DeepSeek-V4-Flash departs from this trend in ways we cannot attribute to capability alone in this case. These trends suggest backbone-dependent preprocessing benefits. Reasoning split: external reasoning scaffolds help mainly non-reasoning backbones. Among the four primary backbones tested, Generation Strategy varies more than any other module. 𝐺 2 (CoT-Enhanced Generation) delivers +4.24 pp on GPT-4o-mini at CPP = 0.026 USD/pp, the second-largest single-module gain after 𝑅1 . The gain shrinks on Gemini-2.5-Flash and GPT-5.4, and turns negative on DeepSeek-V4-Flash (−0.39 pp), with CPP on reasoningcapable backbones one to two orders of magnitude higher. 𝐺 3 (Decomposition-Based Generation) follows a similar pattern, with negligible effect on DeepSeek-V4-Flash; the exception is GPT-5.4, where 𝐺 3 slightly exceeds 𝐺 2 (+1.83 vs. +1.56 pp) at higher CPP. Because DeepSeek-V4-Flash is the only reasoning backbone among the primary four models, we further compare all three generation strategies on Qwen3-235B in thinking mode and o3-mini (Table 2). On Qwen3-235B, 𝐺 2 changes EX by +0.20 pp and 𝐺 3 by −0.52 pp. On o3-mini, the changes are −0.65 and −1.04 pp, with higher cost in every case. Across the three reasoning backbones tested, external reasoning scaffolds provide no consistent accuracy gain over direct generation, despite higher cost. We nevertheless treat this split as suggestive rather than universal, since reasoning style, capability, and model family remain confounded. Pool-bound regime: candidate selection saturates regardless of backbone. All three selectors significantly improve EX over the 𝑁 =5 baseline on every backbone, but we do not test selector against selector; we therefore draw no ranking among 𝐶 1 , 𝐶 2𝑎 , 𝐶 2𝑏 . More consequentially, taking the best selector per backbone, a substantial gap to the oracle ceiling remains everywhere, reaching 5.74 pp on GPT-4o-mini. Candidate selection is recall-bound: once the candidate pool is fixed, further gains must come from improving the pool itself or revising the chosen query post-hoc. The fixed-pool recall ceiling, together with the substantial combined

OUT

Baseline

5,000 4,000 3,000 2,000 1,000 600

1134 1327 1495 463 477 665

10,000

400 200 0

S3a

S3b

C2a

C2b

C2a

C2b

Paradigm DeepSeek-V4-Flash 15,000

OUT

Baseline

10,000

Tokens per question

Backbone

GPT-4o-mini 15,000

Tokens per question

Table 2: Generation strategy results on BIRD dev for two additional reasoning backbones. Qwen3-235B uses thinking mode. Parentheses report ΔEX over 𝐺 1 (pp). Units: Cost in USD, CPP in USD/pp, and IN/OUT in tokens per question.

5,000 4,000 3,000 2,000 1,000 600 400 200 0

S3a

S3b

Paradigm

Figure 5: Per-question input (IN) and output (OUT) token consumption across the 17 paradigm configurations for two backbones, GPT-4o-mini and DeepSeek-V4-Flash, on BIRD. Dashed horizontal lines mark the corresponding baseline token levels for direct comparison across configurations. inference cost of multi-candidate generation and selection, motivates the exclusion of Candidate Selection from both the stacking analysis in § 5 and the cost-aware configuration guideline in § 6.

4.3

Token-Level Resource Decomposition

USD cost alone does not reveal where a paradigm’s resource overhead comes from. Table 1 reports the complete IN/OUT token accounting across all four backbones, while Figure 5 visualizes two contrasting backbones. The source of token overhead closely follows the paradigm mechanism. Example Retrieval and LLM-based Candidate Selection are primarily input-heavy: 𝐸 3 substantially expands the prompt context, while 𝐶 2𝑏 reaches about 15K input tokens. In contrast, Generation Strategy mainly shifts resource use toward output consumption through additional reasoning or decomposition. Schema Linking is more heterogeneous: 𝑆 1 reduces input through schema pruning, whereas 𝑆 4 incurs substantially larger input context from iterative interaction. Refinement shows a different pattern: 𝑅1 remains close to the baseline in both IN and OUT, whereas 𝑅2 and 𝑅3 incur substantially larger token overhead. Across backbones, token consumption varies more strongly in OUT than IN. For a given paradigm, IN is stable across the four backbones, while the two-backbone visualization shows a pronounced OUT gap: DeepSeek-V4-Flash consistently produces substantially more OUT than GPT-4o-mini, with its reasoning traces contributing to the OUT total. This pattern suggests that input demand is more closely tied to pipeline structure, whereas output demand is more sensitive to backbone generation behavior, especially reasoning.

4.4

Single-Module Conclusion

The single-module analysis surfaces one safe default and a structured account of when the rest help and where their resource overhead arises. Only 𝑅1 qualifies as a universal, low-cost

Are These Modules Worth Their Cost? A Paradigm-Level Accuracy-Cost Analysis of In-context Learning Text-to-SQL

Table 3: Cross-module interactions across four backbones. EX is combined execution accuracy (%), with gray values marking combinations that do not outperform the better of the two individual paradigms. Green and red I denote super- and sub-additive interactions. The 𝐺 1 × 𝑅 𝑗 rows are omitted because they reduce to 𝑅 𝑗 and have I = 0 by construction. Cost is reported in USD. All interaction values are measured in percentage points relative to the matched-budget reference baseline for each backbone.

GPT-4o-mini Combination EX↑ 𝐺 2 × 𝑅1 𝐺 2 × 𝑅2 𝐺 2 × 𝑅3 𝐺2 × 𝐸2 𝐺 3 × 𝑅1 𝐺 3 × 𝑅2 𝐺 3 × 𝑅3

DeepSeek-V4-Flash

Total Cost↓ EX↑

56.84 −2.02 55.87 −0.20 57.17 −0.33 54.50 −0.39 55.74 −0.39 52.28 −1.05 52.93 −1.83

0.612 1.143 1.525 0.685 0.657 1.182 1.441

Total Cost↓ EX↑

57.82 +0.07 55.22 −0.97 59.84 +0.26 56.39 −0.07 57.89 −0.07 55.67 −0.72 60.17 +0.39

Cross-Module and -Model Tradeoffs

The single-swap protocol discussed in § 4 isolates each paradigm’s marginal contribution, but two research questions remain unsolved: Q3. When two paradigms are used together, do their gains compose additively, or does their combination yield smaller or larger gains than expected across different backbones? Q4. Given a fixed budget, is it more cost-efficient to engineer a richer pipeline on a cheaper backbone, or instead to simply upgrade to a stronger backbone running a lean pipeline?

5.1

Module Interactions

We study Q3 through a systematic analysis of module interactions. Since exhaustive coverage is infeasible, we focus on the full 3 × 3 Generation × Refinement grid across all backbones. We choose this pair because refinement revises the SQL produced by generation, making their interaction particularly direct. For paradigms 𝐴 and 𝐵, we quantify their interaction in percentage points as I = ΔEXpp (𝐴&𝐵) − ΔEXpp (𝐴) − ΔEXpp (𝐵),

(9)

where positive I indicates super-additive gains and negative I indicates sub-additive gains, both relative to the baseline pipeline. Table 3 reports the resulting configurations and interaction terms. Generation × Refinement. Of the 24 nontrivial 𝐺 2 /𝐺 3 × 𝑅1–3 configurations, 17 achieve higher EX than the better individual paradigm. The pattern is strongest on GPT-5.4, where six combinations improve over both individual paradigms; five of six do so on both GPT-4o-mini and Gemini-2.5-Flash. DeepSeek-V4-Flash is the exception, with only one of six combinations improving over both paradigms. This is consistent with the single-module results in § 4: neither 𝐺 2 nor 𝐺 3 outperforms 𝐺 1 on this reasoning backbone, so the resulting combinations rarely outperform refinement alone.

65.12 −0.07 63.62 −0.20 66.04 +0.13 64.34 +0.20 63.43 +0.19 59.45 −2.42 64.60 +0.64

0.743 1.457 2.004 0.793 0.733 1.501 2.067

default; other modules’ contributions are backbone-dependent. The dependence has structure: input-side scaffolding tracks capability, generation-side scaffolding tracks reasoning style, and post-hoc selection saturates against a pool-bound ceiling. Resource use shows a complementary pattern: IN is more closely tied to pipeline structure, whereas OUT is more sensitive to backbone generation behavior. § 5 next asks whether these effects compose when paradigms are stacked, and how a fixed budget should be allocated across stages.

Gemini-2.5-Flash

GPT-5.4

Total Cost↓ EX↑ 1.971 3.856 4.748 2.217 1.956 4.523 5.304

Total Cost↓

64.60 −0.26 64.73 +1.11 65.19 −0.39 65.32 +0.07 65.38 +0.26 64.47 +0.59 66.56 +0.72

10.310 24.751 32.632 12.190 12.114 26.360 34.801

Table 4: Module-best pipelines on BIRD dev. Parentheses show ΔEX (pp) over the 𝑁 =1 baseline; API costs are in USD. Stacks follow the cost-aware rule in § 5.2 and favor paradigms with low CPP and broadly positive gains across tested models.

Backbone

Stack

GPT-4o-mini 𝐸 2 +𝑆 2 +𝐺 2 +𝑅1 DeepSeek-V4-Flash 𝐸 2 +𝑆 2 +𝐺 1 +𝑅1 Gemini-2.5-Flash 𝐸 2 +𝑆 2 +𝐺 2 +𝑅1 GPT-5.4 𝐸 2 +𝑆 2 +𝐺 2 +𝑅1

EX (%)↑

Cost↓

58.02 (+8.28) 0.759 61.15 (+4.82) 1.584 67.14 (+7.30) 2.916 69.43 (+8.15) 16.237

The interaction terms reveal how these gains compose across backbones. On GPT-4o-mini, stacking usually outperforms the better individual paradigm, but all six interactions are sub-additive. This may reflect the weaker backbone’s limited capacity to exploit generation and refinement jointly, yielding overlapping gains across stages. GPT-5.4 exhibits the opposite pattern: all six combinations outperform both individual paradigms, and four are super-additive. Generation × Example Retrieval. We further examine whether this backbone-dependent pattern extends to the input side through 𝐺 2 ×𝐸 2 . The combination outperforms both individual paradigms on the three non-reasoning backbones, but not on DeepSeek-V4-Flash, where 𝐺 2 does not improve the baseline in isolation. I remains small across all four backbones, ranging from −0.39 to +0.20 pp, indicating near-additive composition for this module pairing overall. Overall pattern. Results from the two module pairings show that interaction effects differ systematically across backbones. Module stacking improves accuracy on most backbones, but how its gains compose varies with backbone capability. Since results from two-paradigm combinations do not determine the performance of complete multi-module stacks, § 5.2 evaluates backbonespecific stacks and reports end-to-end accuracy and cost jointly.

5.2

Stacking vs. Upgrading

Stack construction rule. We exclude Candidate Selection because its gains are bounded by candidate-pool diversity (§ 4.2) and do not justify sampling 𝑁 >1 candidates from a single generator. For each backbone and each remaining module, we select the nontrivial paradigm with the lowest CPP whose EX gain is positive on that

Execution Accuracy (%)

Jiayan Lin, Yujia Liu, Zijin Hong, Zheng Yuan, Yilin Xiao, Hao Chen, Qinggang Zhang, Xiao Huang, and Feiran Huang

Table 5: Guideline validation on additional backbones. EX (%) and total API cost in USD are measured on BIRD; parenthesized values report ΔEX in percentage points over Tier 1. T1–T3 denote three progressively richer configurations, ranging from the lean baseline to the full cost-aware pipeline.

EX (%) Baseline (N = 1) Module-best pipeline

75 0.1

0.2

0.5

Total API Cost on Spider dev (USD, log scale)

Figure 6: Cost–EX tradeoffs on Spider dev. Hollow circles denote 𝑁 =1 baselines, filled stars denote the module-best pipelines selected on BIRD and applied unchanged to Spider. The horizontal axis reports total cost on a logarithmic scale. backbone and on a majority of backbones, retaining the trivial paradigm if none qualifies. This selects 𝑆 2 rather than the cheaper 𝑆 1 on GPT-4o-mini: although 𝑆 1 gains +0.13 pp on that backbone, its gain is negative on the other three. The resulting stacks in Table 4 largely follow the marginal analysis: all use 𝑅1 , while DeepSeek-V4Flash retains 𝐺 1 because neither nontrivial generation paradigm improves its EX. The only substantive departure from the marginalEX ranking occurs in GPT-5.4’s generation module, where the rule selects 𝐺 2 for its lower CPP despite 𝐺 3 ’s slightly higher EX overall. BIRD tradeoff. Table 4 summarizes the resulting stacks on BIRD. Stacking improves EX on every backbone, but by different margins: GPT-4o-mini, Gemini-2.5-Flash, and GPT-5.4 each gain more than 7 pp, while DeepSeek-V4-Flash gains noticeably less, consistent with its weaker response to pipeline engineering observed in § 4. At moderate budgets, engineering a richer pipeline on a mid-tier backbone outperforms upgrading to a frontier backbone running lean: Gemini-2.5-Flash’s stack Pareto-dominates GPT5.4’s lean baseline, achieving substantially higher EX at lower cost. This reframes choice as a backbone–pipeline tradeoff rather than a preference for the strongest model. At the high-budget end, GPT-5.4’s stack attains the highest EX, but its 2.29 pp advantage over Gemini-2.5-Flash’s stack comes at more than 5× the total cost. Spider generalization. To test cross-benchmark transfer, we apply the BIRD-derived stacks unchanged to Spider. As shown in Figure 6, all four stacks improve EX over their respective lean baselines, with gains ranging from +3.68 to +6.77 pp. The backbone–pipeline tradeoff persists on Spider: the DeepSeek-V4-Flash and Gemini2.5-Flash stacks both Pareto-dominate GPT-5.4’s lean baseline. The GPT-4o-mini stack nearly matches that baseline (80.17% vs. 80.27%) at $0.315 rather than $1.879, while the DeepSeek-V4-Flash and Gemini-2.5-Flash stacks surpass it, reaching 82.01% and 84.72% at lower costs of $0.679 and $1.172, respectively. GPT-5.4’s stack achieves the highest EX of 87.04%, but its 2.32 pp advantage over Gemini-2.5-Flash’s stack requires 5.62× the total cost overall.

5.3

Cross-Module and -Model Conclusion

Cross-module composition is backbone-dependent, and at moderate budgets the cost-efficient frontier runs through stacked mid-tier pipelines rather than lean configurations on frontier

Backbone

Llama-3.1-8B-Instruct 37.61 40.22 (+2.61) Qwen3-32B 50.59 52.80 (+2.21) o3-mini 54.11 55.87 (+1.76) Qwen3-235B 57.82 59.06 (+1.24) Claude Sonnet 4.5 62.71 65.25 (+2.54)

Cost T3 44.07 (+6.46) 55.48 (+4.89) 59.97 (+5.87) 61.80 (+3.98) 70.34 (+7.63)

0.05 0.06 0.09 0.43 0.52 0.93 5.67 6.40 12.67 4.55 5.76 11.36 8.62 9.65 18.94

backbones. Marginal results remain useful for stack construction, but interactions shape how module gains combine. Applying the BIRD-derived stacks unchanged to Spider shows that the resulting backbone–pipeline tradeoff extends beyond BIRD. Table 4 reports a separate stack for each backbone; § 6 distills these results into a unified guideline and evaluates it on five diverse additional models.

Cost-Efficient Configuration Guideline

The per-backbone stacks in § 5.2 are backbone-specific selections derived from per-paradigm evaluation and therefore cannot be assumed in advance when configuring a new system. The practical question is instead whether the recurring patterns across these stacks can be compressed into a small set of configurations that requires only coarse knowledge of the backbone, rather than a new paradigm search for every model. This section distills the preceding single-module and stacking results into such a prescriptive guideline, examines the empirical patterns behind its recommendations, and directly evaluates its transfer to five additional validation backbones spanning diverse capability levels and reasoning behaviors.

6.1

From Stacks to a Tiered Guideline

We distill the single-module and stacking results in our study into three clear incremental tiers for practical use. Tier 1 is the bare backbone running a single-pass baseline. Tier 2 adds 𝑅1 (ExecutionFeedback Refinement), the only paradigm that improves all four primary backbones at consistently low cost. Tier 3 further adds 𝐸 2 (Masked Question Similarity) and 𝑆 2 (Pseudo-SQL Guided Linking), together with a backbone-conditional generation strategy: 𝐺 2 (CoT-Enhanced) for non-reasoning backbones and 𝐺 1 (trivial) for reasoning-capable ones, following the split identified in § 4.2. These choices are not obtained by simply taking the highestaccuracy paradigm in each module. They preserve the components that recur across the cost-efficient stacks in § 5.2. All four stacks contain 𝐸 2 , 𝑆 2 , and 𝑅1 , whereas Generation Strategy is the main component that changes with backbone reasoning behavior: the three non-reasoning stacks use 𝐺 2 , while DeepSeek-V4-Flash retains 𝐺 1 . This recurring structure in turn motivates a fixed tier definition with the generation choice conditioned on the backbone. The resulting guideline recommends a tier from the backbone profile and accuracy–cost preference, without requiring perparadigm search for a new backbone. Figure 7 summarizes the resulting tier construction and backbone-conditioned selection rule.

Are These Modules Worth Their Cost? A Paradigm-Level Accuracy-Cost Analysis of In-context Learning Text-to-SQL

[Tier Construction] Bare-backbone baseline. Use the single-pass baseline with all modules at their trivial settings. ↩→ T1: Bare backbone (𝑁 =1). Universal refinement rule. 𝑅1 is the only paradigm that improves all tested backbones at consistently low cost. ↩→ T2: T1 + 𝑅1 . Backbone-conditioned stacking rule. 𝐸 2 + 𝑆 2 persist across backbone stacks, while generation follows the reasoning split: ( 𝐺 2 , non-reasoning backbone, ∗ 𝐺 = 𝐺 1 , reasoning backbone. ↩→ T3: T2 + 𝐸 2 + 𝑆 2 + 𝐺 ∗ . [Candidate Selection] Cost-efficient setting. Candidate Selection faces a fixed-pool recall ceiling and 𝑁 >1 sampling overhead, so it is excluded from the costefficient tiers. Accuracy-first option. When cost is secondary, Candidate Selection may be added for further accuracy, subject to the candidate-pool recall ceiling. [Tier Selection] Reasoning Small / mid-tier non-reasoning Frontier

T2 is the cost-efficient default; consider T3 when extra accuracy justifies the cost. Prefer T3. Use T3 when maximum EX is prioritized; otherwise consider a cheaper backbone with T3.

Figure 7: Evidence-backed cost-efficient configuration guideline distilled from the single-module and stacking analyses. The three tiers progressively incorporate paradigms supported by robust accuracy–cost evidence, while Candidate Selection is treated separately due to its sampling overhead and fixed-pool recall ceiling. The recommended tier further depends on backbone capability and reasoning behavior.

6.2

Patterns Behind the Guideline

Pipeline engineering is most cost-effective on small and midtier non-reasoning backbones. The Tier 1 to Tier 3 improvement is largest on the smallest backbone in our study, while a mid-tier non-reasoning backbone at Tier 3 costs a fraction of a frontier backbone at Tier 1 (§ 5.2). For example, Gemini-2.5-Flash at Tier 3 reaches higher EX than GPT-5.4 at Tier 1 while costing substantially less in our setting. The guideline therefore favors Tier 3 for these backbones, while using Tier 2 as the cost-efficient default for reasoning backbones and reserving Tier 3 on frontier backbones for accuracy-first settings; § 6.3 tests this pattern on five backbones.

6.3

Validation on Additional Backbones

To test whether the guideline transfers beyond the four primary backbones from which its tier structure was derived, we directly apply the three prescribed tiers to five additional validation backbones spanning different capability levels and reasoning behaviors,

without conducting any per-paradigm search or adapting the tier composition to individual models. All five validation backbones consistently improve from Tier 1 to both Tier 2 and Tier 3. As shown in Table 5, Tier 2 gains range from +1.24 to +2.61 pp, while Tier 3 gains range from +3.98 to +7.63 pp. The improvement is observed across capability levels and both non-reasoning and reasoning settings, indicating that the fixed tier compositions continue to transfer beyond the four primary backbones used for stack construction. The corresponding costs also follow the intended tier ordering, with Tier 2 adding limited overhead and Tier 3 trading a larger cost increase for higher EX across five validation backbones.

Related Work

Empirical Analysis of Text-to-SQL. Several studies analyze specific components or configurations of Text-to-SQL systems. Li et al. [17] introduce NL2SQL360, a multi-angle evaluation framework that compares NL2SQL methods across data domains, SQL characteristics, and evaluation settings, and further explores the selection of solutions for different application needs. At the component level, Maamari et al. [22] revisit explicit schema linking and show that stronger LLMs can often operate directly over the full schema when it fits within the context window, avoiding errors caused by filtering necessary elements. In contrast, Yuan et al. [42] improve schema linking by formulating the selection of relevant schema elements as a constrained knapsack optimization. For generation, Tai et al. [31] also show that iterative least-to-most prompting can be unnecessary and that overly detailed reasoning may propagate errors. Recent systems increasingly combine multiple such mechanisms into elaborate inference-time pipelines. CHASE-SQL [24] combines multi-path candidate generation with preference-based pairwise selection; DeepEye-SQL [16] organizes linking, diverse generation, verification, and candidate selection through a workflow; and Agentar-Scale-SQL [34] orchestrates internal reasoning, iterative refinement, and parallel generation and selection for test-time scaling. These systems demonstrate the benefit of increasingly sophisticated pipeline orchestration, but still primarily optimize integrated system designs and evaluate components within their own pipelines. Our study complements these efforts by isolating each module’s marginal accuracy–cost contribution under one controlled implementation across multiple backbones; the schema-linking and CoT trends further extend prior observations to this cost-aware setting. Cost-Aware LLM Systems. Cost-aware LLM deployment has been studied at three complementary layers. At the model-selection layer, FrugalGPT [3], SATER [28], and RouteLLM [23] use routing or cascading to navigate tradeoffs between performance and cost. At the inference-resource layer, LLMLingua [14] and Sketch-of-Thought [1] reduce prompt or reasoning token usage, while recent work studies compute-optimal and budget-aware allocation of test-time reasoning resources [21, 30]. At the workflow layer, CATP LLM [37] explicitly balances task performance against tool-execution cost. Within Text-to-SQL, Dönder et al. [5] propose N-rep, which reduces inference cost through multiple schema representations without CoT or self-consistency. These efforts optimize model choice, inference resources, or workflow execution; we instead directly ask which modules of the Text-to-SQL pipeline justify their added cost.

Jiayan Lin, Yujia Liu, Zijin Hong, Zheng Yuan, Yilin Xiao, Hao Chen, Qinggang Zhang, Xiao Huang, and Feiran Huang

Conclusion

We studied the ICL Text-to-SQL pipeline as a modular object in its own right, isolating the marginal contribution and cost of each design paradigm under a unified controlled framework across four backbones. Execution-feedback refinement is the only paradigm that improves all four primary backbones at consistently low cost, making it a safe first addition; most others pay off only under backbone-dependent conditions. Token accounting suggests that input demand is closely tied to pipeline structure, whereas output demand is sensitive to backbone generation behavior. Building on this analysis, we found that a fixed budget is often better spent engineering a richer pipeline over a mid-tier backbone than upgrading to a frontier model with a lean pipeline. The BIRD-derived stacks also transfer to Spider, where all four improve accuracy and the backbone–pipeline tradeoff persists. We distill these observations into a backbone-dependent, tiered guideline that transfers to five additional validation backbones, offering practitioners a more practical, cost-aware starting point for configuring ICL Text-to-SQL systems than aggregate accuracy alone in practical settings.

Ethical Considerations

This work evaluates text-to-SQL methods on the publicly available BIRD benchmark [19]. Nevertheless, deployed text-to-SQL systems, especially in real world database applications, can expose sensitive database contents or execute harmful queries when model outputs are incorrect or maliciously prompted. Practical deployments should therefore enforce least-privilege access, use read-only execution whenever possible, validate generated SQL, sandbox query execution, and maintain proper human oversight and audit logs.

References [1] Simon A. Aytes, Jinheon Baek, and Sung Ju Hwang. 2025. Sketch-of-Thought: Efficient LLM Reasoning with Adaptive Cognitive-Inspired Sketching. In Empirical Methods in Natural Language Processing (EMNLP). [2] Zhenbiao Cao, Yuanlei Zheng, Zhihao Fan, Xiaojin Zhang, Wei Chen, and Xiang Bai. 2024. RSL-SQL: Robust Schema Linking in Text-to-SQL Generation. arXiv preprint arXiv:2411.00073 (2024). [3] Lingjiao Chen, Matei Zaharia, and James Zou. 2024. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. Transactions on Machine Learning Research (TMLR) (2024). [4] Minghang Deng, Ashwin Ramachandran, Canwen Xu, Lanxiang Hu, Zhewei Yao, Anupam Datta, and Hao Zhang. 2025. ReFoRCE: A Text-to-SQL Agent with Self-Refinement, Format Restriction, and Column Exploration. In ICLR 2025 Workshop: VerifAI: AI Verification in the Wild. [5] Yusuf Denizay Dönder, Derek Hommel, Andrea W Wen-Yi, David Mimno, and Unso Eun Seo Jo. 2025. Cheaper, Better, Faster, Stronger: Robust Text-to-SQL without Chain-of-Thought or Fine-Tuning. arXiv preprint arXiv:2505.14174 (2025). [6] Xuemei Dong, Chao Zhang, Yuhang Ge, Yuren Mao, Yunjun Gao, Lu Chen, Jinshu Lin, and Dongfang Lou. 2023. C3: Zero-shot Text-to-SQL with ChatGPT. arXiv:2307.07306 (2023). [7] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024). [8] Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. 2024. Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation. In International Conference on Very Large Data Bases (VLDB). [9] Yingqi Gao, Yifu Liu, Xiaoxia Li, Xiaorong Shi, Yin Zhu, Yiming Wang, Shiqi Li, Wei Li, Yuntao Hong, Zhiling Luo, Jinyang Gao, Liyu Mou, and Yu Li. 2024. Xiyan-sql: A multi-generator ensemble framework for text-to-sql. arXiv preprint arXiv:2411.08599 (2024). [10] Zijin Hong, Hao Chen, Zheng Yuan, Qinggang Zhang, Luyao Zhuang, Qing Liao, Feiran Huang, Yangqiu Song, and Xiao Huang. 2026. ErrorLLM: Modeling SQL Errors for Text-to-SQL Refinement. In Conference on Knowledge Discovery and Data Mining (KDD).

[11] Zijin Hong, Hao Wu, Su Dong, Junnan Dong, Yilin Xiao, Yujing Zhang, Zhu Wang, Feiran Huang, Linyi Li, Hongxia Yang, and Xiao Huang. 2026. Benchmarking LLMs’ mathematical reasoning with unseen random variables questions. In Conference on Artificial Intelligence (AAAI). [12] Zijin Hong, Zheng Yuan, Hao Chen, Qinggang Zhang, Feiran Huang, and Xiao Huang. 2024. Knowledge-to-SQL: Enhancing SQL Generation with Data Expert LLM. In Findings of Association for Computational Linguistics (ACL). [13] Zijin Hong, Zheng Yuan, Qinggang Zhang, Hao Chen, Junnan Dong, Feiran Huang, and Xiao Huang. 2025. Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQL. IEEE Transactions on Knowledge and Data Engineering (TKDE) (2025). [14] Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models. In Empirical Methods in Natural Language Processing (EMNLP). [15] Dongjun Lee, Choongwon Park, Jaehyuk Kim, and Heesoo Park. 2025. MCS-SQL: Leveraging Multiple Prompts and Multiple-Choice Selection For Text-to-SQL Generation. In International Conference on Computational Linguistics (COLING). [16] Boyan Li, Chong Chen, Zhujun Xue, Yinan Mei, and Yuyu Luo. 2026. DeepEyeSQL: A software-engineering-inspired text-to-sql framework. In Conference on Management of Data (SIGMOD). [17] Boyan Li, Yuyu Luo, Chengliang Chai, Guoliang Li, and Nan Tang. 2024. The Dawn of Natural Language to SQL: Are We Fully Ready?. In International Conference on Very Large Data Bases (VLDB). [18] Boyan Li, Jiayi Zhang, Ju Fan, Yanwei Xu, Chong Chen, Nan Tang, and Yuyu Luo. 2025. Alpha-SQL: Zero-Shot Text-to-SQL using Monte Carlo Tree Search. In International Conference on Machine Learning (ICML). [19] Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2023. Can LLM Already Serve as A Database Interface? A Big Bench for Large-Scale Database Grounded Text-to-SQLs. In Advances in Neural Information Processing Systems (NeurIPS). [20] Zhishuai Li, Xiang Wang, Jingjing Zhao, Sun Yang, Guoqing Du, Xiaoru Hu, Bin Zhang, Yuxiao Ye, Ziyue Li, Rui Zhao, and Hangyu Mao. 2025. Pet-sql: A prompt-enhanced two-round refinement of text-to-sql with cross-consistency. In International Conference on Database Systems for Advanced Applications (DASFAA). [21] Junhong Lin, Xinyue Zeng, Jie Zhu, Song Wang, Julian Shun, Jun Wu, and Dawei Zhou. 2026. Plan and Budget: Effective and Efficient Test-Time Scaling on Reasoning Large Language Models. In International Conference on Learning Representations (ICLR). [22] Karime Maamari, Fadhil Abubaker, Daniel Jaroslawicz, and Amine Mhedhbi. 2024. The Death of Schema Linking? Text-to-SQL in the Age of Well-Reasoned Language Models. In NeurIPS 2024 Third Table Representation Learning Workshop (NeurIPS). [23] Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E Gonzalez, M Kadous, and Ion Stoica. 2025. RouteLLM: Learning to Route LLMs from Preference Data. In International Conference on Learning Representations (ICLR). [24] Mohammadreza Pourreza, Hailong Li, Ruoxi Sun, Yeounoh Chung, Shayan Talaei, Gaurav Tarlok Kakkar, Yu Gan, Amin Saberi, Fatma Ozcan, and Sercan O Arik. 2025. Chase-sql: Multi-path reasoning and preference optimized candidate selection in text-to-sql. In International Conference on Learning Representations (ICLR). [25] Mohammadreza Pourreza and Davood Rafiei. 2023. Din-sql: Decomposed incontext learning of text-to-sql with self-correction. In Advances in Neural Information Processing Systems (NeurIPS). [26] Ge Qu, Jinyang Li, Bowen Qin, Xiaolong Li, Nan Huo, Chenhao Ma, and Reynold Cheng. 2025. SHARE: An SLM-based Hierarchical Action CorREction Assistant for Text-to-SQL. In Association for Computational Linguistics (ACL). [27] Tonghui Ren, Yuankai Fan, Zhenying He, Ren Huang, Jiaqi Dai, Can Huang, Yinan Jing, Kai Zhang, Yifan Yang, and X. Sean Wang. 2024. PURPLE: Making a Large Language Model a Better SQL Writer. In International Conference on Data Engineering (ICDE). [28] Yuanzhe Shen, Yide Liu, Zisu Huang, Ruicheng Yin, Xiaoqing Zheng, and Xuanjing Huang. 2025. SATER: A Self-Aware and Token-Efficient Approach to Routing and Cascading. In Empirical Methods in Natural Language Processing (EMNLP). [29] Jie Shi, Bo Xu, Jiaqing Liang, Yanghua Xiao, Jia Chen, Chenhao Xie, Peng Wang, and Wei Wang. 2025. Gen-SQL: Efficient Text-to-SQL By Bridging Natural Language Question And Database Schema With Pseudo-Schema. In International Conference on Computational Linguistics (COLING). [30] Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2025. Scaling LLM Test-Time Compute Optimally Can be More Effective than Scaling Parameters for Reasoning. In International Conference on Learning Representations (ICLR). [31] Chang-Yu Tai, Ziru Chen, TIANSHU ZHANG, Xiang Deng, and Huan Sun. 2023. Exploring Chain of Thought Style Prompting for Text-to-SQL. In Empirical Methods in Natural Language Processing (EMNLP).

Are These Modules Worth Their Cost? A Paradigm-Level Accuracy-Cost Analysis of In-context Learning Text-to-SQL

[32] Shayan Talaei, Mohammadreza Pourreza, Yu-Chen Chang, Azalia Mirhoseini, and Amin Saberi. 2024. CHESS: Contextual Harnessing for Efficient SQL Synthesis. arXiv:2405.16755 (2024). [33] Bing Wang, Changyu Ren, Jian Yang, Xinnian Liang, Jiaqi Bai, LinZheng Chai, Zhao Yan, Qian-Wen Zhang, Di Yin, Xing Sun, and Zhoujun Li. 2025. MACSQL: A Multi-Agent Collaborative Framework for Text-to-SQL.. In International Conference on Computational Linguistics (COLING). [34] Pengfei Wang, Baolin Sun, Xuemei Dong, Yaxun Dai, Hongwei Yuan, Mengdie Chu, Yingqi Gao, Xiang Qi, Peng Zhang, and Ying Yan. 2025. Agentar-Scale-SQL: Advancing Text-to-SQL through Orchestrated Test-Time Scaling. arXiv preprint arXiv:2509.24403 (2025). [35] Yihan Wang, Peiyu Liu, and Xin Yang. 2025. LinkAlign: Scalable Schema Linking for Real-World Large-Scale Multi-Database Text-to-SQL. In Empirical Methods in Natural Language Processing (EMNLP). [36] Ziyang Wang, Yuanlei Zheng, Zhenbiao Cao, Xiaojin Zhang, Zhongyu Wei, Pei Fu, Zhenbo Luo, Wei Chen, and Xiang Bai. 2026. AutoLink: Autonomous Schema Exploration and Expansion for Scalable Schema Linking in Text-to-SQL at Scale. In Conference on Artificial Intelligence (AAAI). [37] Duo Wu, Jinghe Wang, Yuan Meng, Yanning Zhang, Le Sun, and Zhi Wang. 2025. CATP-LLM: Empowering Large Language Models for Cost-Aware Tool Planning. In International Conference on Computer Vision (ICCV). [38] Xiangjin Xie, Guangwei Xu, Lingyan Zhao, and Ruijie Guo. 2025. OpenSearchSQL: Enhancing Text-to-SQL with Dynamic Few-shot and Consistency Alignment. In Conference on Management of Data (SIGMOD). [39] Yuanzhen Xie, Xinzhou Jin, Tao Xie, Mingxiong Lin, Liang Chen, Chenyun Yu, Lei Cheng, Chengxiang Zhuo, Bo Hu, and Zang Li. 2024. Decomposition for Enhancing Attention: Improving LLM-based Text-to-SQL through Workflow Paradigm. In Findings of Association for Computational Linguistics (ACL). [40] Hongyu Yao, Zijin Hong, Hao Chen, Zhiqing Li, Qijie Shen, Zuobin Ying, Qihua Feng, Huan Gong, and Feiran Huang. 2026. Macro Graph of Experts for BillionScale Multi-Task Recommendation. In Conference on Knowledge Discovery and Data Mining (KDD). [41] Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task. In Empirical Methods in Natural Language Processing (EMNLP). [42] Zheng Yuan, Hao Chen, Zijin Hong, Qinggang Zhang, Feiran Huang, Qing Li, and Xiao Huang. 2025. Knapsack Optimization-based Schema Linking for LLM-based Text-to-SQL Generation. In International Conference on Data Engineering (ICDE).

Jiayan Lin, Yujia Liu, Zijin Hong, Zheng Yuan, Yilin Xiao, Hao Chen, Qinggang Zhang, Xiao Huang, and Feiran Huang

Table 6: Per-backbone EX (%) on BIRD dev (1,534 examples). ΔEX [95% CI] is the percentage-point gain over the matched-budget baseline (𝑁 =5 for Candidate Selection; 𝑁 =1 otherwise), and CPP is incremental USD/pp. “–” denotes undefined CPP, “∗ ” Pareto domination, and the / keys mark the best/second best results. GPT-4o-mini

EX ↑

DeepSeek-V4-Flash

Gemini-2.5-Flash

GPT-5.4

[95% CI]

CPP ↓

EX ↑

𝚫EX ↑

[95% CI]

CPP ↓

EX ↑

𝚫EX ↑

[95% CI]

CPP ↓

EX ↑

𝚫EX ↑

[95% CI]

CPP ↓

E1 50.07 E2 50.65 E3 49.80 Schema Linking

+0.33 [-0.01, +0.67] +0.91 [+0.26, +1.56] +0.07 [-0.16, +0.29]

0.207 0.144 7.259

56.06 56.84 57.11

-0.26 [-0.57, +0.05] +0.52 [+0.01, +1.03] +0.78 [+0.06, +1.50]

– 0.326 0.660

60.43 61.28 60.10

+0.59 [+0.06, +1.12] +1.43 [+0.74, +2.13] +0.26 [-0.18, +0.70]

0.401 0.417 5.317

62.58 63.69 63.49

+1.30 [+0.63, +1.97] +2.41 [+1.42, +3.40] +2.22 [+1.28, +3.15]

0.995 0.957 3.417

S1 49.87 S2 52.15 S3𝑎 51.11 S3𝑏 51.83 S4 52.93 SQL Generation

+0.13 +2.41 +1.37 +2.09 +3.19

[-0.13, +0.39] [+1.49, +3.33] [+0.69, +2.05] [+1.23, +2.95] [+2.11, +4.27]

– 0.075 0.109 0.090 0.271

55.93 57.04 56.19 55.87 56.52

-0.39 +0.72 -0.13 -0.46 +0.20

[-0.75, -0.03] [+0.08, +1.36] [-0.39, +0.13] [-0.88, -0.04] [-0.09, +0.49]

– 0.491 – – 4.262

59.13 61.15 60.89 60.76 61.34

-0.72 +1.30 +1.04 +0.91 +1.50

[-1.27, -0.16] [+0.63, +1.98] [+0.42, +1.67] [+0.32, +1.51] [+0.79, +2.21]

– 0.575 0.599 0.847 1.224

60.63 62.39 60.82 59.97 62.06

-0.65 +1.11 -0.46 -1.30 +0.78

[-1.19, -0.11] [+0.45, +1.77] [-0.95, +0.03] [-1.98, -0.63] [+0.21, +1.35]

– 3.293 – – 20.870

G1 49.74 0.00 G2 53.98 +4.24 [+2.96, +5.52] G3 51.24 +1.50 [+0.77, +2.23] Candidate Selection

– 0.026 0.093

56.32 55.93 56.13

0.00 -0.39 [-0.75, -0.03] -0.20 [-0.49, +0.09]

– – –

59.84 62.71 60.76

0.00 +2.87 [+1.80, +3.94] +0.91 [+0.32, +1.51]

– 0.170 0.976

61.28 62.84 63.10

0.00 +1.56 [+0.80, +2.32] +1.83 [+1.00, +2.65]

– 1.723 2.285

𝚫EX ↑

Example Retrieval

C1 51.11 C2𝑎 50.52 C2𝑏 51.17 Refinement R1 R2 R3

54.63 51.83 53.26

∗

57.30 59.00 58.54

+0.85 [+0.19, +1.51] +2.54 [+1.47, +3.61] +2.09 [+1.14, +3.04]

0.333 0.953

60.89 61.60 61.93

+0.85 [+0.27, +1.43] +1.56 [+0.80, +2.32] +1.89 [+1.04, +2.74]

1.806 2.858

62.71 63.82 63.89

+1.30 [+0.63, +1.97] +2.41 [+1.42, +3.40] +2.48 [+1.48, +3.48]

7.249 15.975

+4.89 [+3.53, +6.25] +2.09 [+1.23, +2.95] +3.52 [+2.38, +4.66]

0.016 0.305 0.276

58.15 56.58 59.97

+1.83 [+0.99, +2.67] +0.26 [-0.05, +0.57] +3.65 [+2.47, +4.83]

0.040 2.708 0.365

62.32 60.95 63.04

+2.48 [+1.48, +3.48] +1.11 [+0.45, +1.77] +3.19 [+2.08, +4.30]

0.026 1.964 1.016

63.30 62.06 64.02

+2.02 [+1.15, +2.89] +0.78 [+0.21, +1.35] +2.74 [+1.68, +3.80]

0.297 18.680 7.769

Full Taxonomy Mapping

Multi-paradigm cells. A method may instantiate multiple paradigms within one module. This occurs mainly in Generation Strategy, where systems combine direct, CoT-enhanced, and decompositionbased generators, and in Schema Linking, where retrieval-based and LLM-based signals are combined before SQL generation. Classification protocol. We classify each method according to the primary system reported in the original paper, excluding ablation variants. When the paradigm is not stated explicitly, we infer it from the provided prompts or pseudocode; if neither provides sufficient evidence, the cell is marked as – in the final taxonomy table.

Paradigm Implementation Details

This section details the controlled implementations of the 17 paradigmlevel configurations evaluated in the main paper. We organize them into pre-processing (Example Retrieval and Schema Linking; Figure 8), SQL generation (Generation Strategy; Figure 9), and postprocessing (Candidate Selection and Refinement; Figure 10). Box layout. Each module card specifies its shared inputs, backbone, and fallback behavior; maps paradigms to mechanisms and per-query LLM calls; notes auxiliary calls or behavior and their cost treatment; and identifies the trivial baseline used in the main marginal analysis for controlled comparison across paradigms..

∗

0.771 1.389

Table 7 maps 24 ICL Text-to-SQL methods to paradigms across five pipeline modules. A checkmark indicates documented use of a paradigm, while a dash indicates that such use is not established in the original paper. For Direct LLM Linking and LLM-Based Preference Selection, sub-paradigm distinctions (𝑆 3𝑎 vs. 𝑆 3𝑏 and 𝐶 2𝑎 vs. 𝐶 2𝑏 ) are recorded within cells rather than assigned separate columns.

∗

+1.43 [+0.74, +2.13] +0.85 [+0.36, +1.34] +1.50 [+0.79, +2.21]

API Pricing

All USD costs reported in the main paper and this supplement are calculated using the API prices in Table 8, recorded on May 2, 2026.

For each benchmark, the total configuration cost is the sum of inputand output-token charges over its full development split. Table 8: API prices in USD per million tokens, recorded on May 2, 2026. For reasoning backbones, output charges include all billed reasoning tokens used in evaluation. Model GPT-4o-mini Gemini-2.5-Flash DeepSeek-V4-Flash GPT-5.4 Llama-3.1-8B-Instruct Qwen3-32B o3-mini Qwen3-235B-A22B (Thinking) Claude Sonnet 4.5

Input

Output

0.15 0.30 0.14 2.50 0.02 0.08 1.10 0.23 3.00

0.60 2.50 0.28 15.00 0.04 0.28 4.40 2.30 15.00

Detailed Results with Confidence Intervals

This section supplements the module-level results in Table 2 of the main paper. For every paradigm–backbone pair, Table 6 reports execution accuracy, its change from the matched-budget reference baseline with a corresponding 95% confidence interval, and cost per EX point (CPP). The main-paper table provides the corresponding API costs and mean per-question input/output-token counts. Confidence intervals. For each paradigm–backbone pair, we compare per-example correctness with a matched-budget reference over the 1,534 BIRD development examples. Two-sided 95% confidence intervals for ΔEX use the normal approximation to the paired difference based on discordant outcome counts, while significance is assessed by a two-sided McNemar’s test at 𝛼=0.05. Candidate Selection uses an 𝑁 =5 reference, whereas all other paradigms use 𝑁 =1, isolating the paradigm effect from additional candidate sampling. We report the intervals directly to preserve effect direction and magnitude rather than only binary significance decisions.

Are These Modules Worth Their Cost? A Paradigm-Level Accuracy-Cost Analysis of In-context Learning Text-to-SQL

Table 7: Taxonomy of representative Text-to-SQL methods.

RetrievalBased Linking

✓

–

Pseudo-SQL Guided Linking

Single-shot Direct Linking

– –

✓

–

✓

CoT-Enhanced Generation

✓

DecompositionBased Generation

✓

ExecutionBased Voting

Pairwise Comparison

LLM-Based Preference Selection

–

✓

ExecutionFeedback Refinement

–

SelfCorrection

✓

–

Agentic Refinement

Refinement

–

Candidate Selection

SQL Skeleton Similarity

✓

Generation Strategy

Masked Question Similarity

–

Schema Linking

–

Example Retrieval

Question Similarity

✓

–

Venue

–

Method

AAAI’26

–

Direct Generation

SIGMOD’26

–

Agentic Linking

AutoLink

–

Direct LLM Linking

DeepEye-SQL –

–

✓

–

✓

–

AAAI’25

–

✓

–

MAGIC

Pairwise Comparison

✓

–

✓

Scoring and Ranking

✓ –

–

✓ –

–

✓ ✓

–

– –

✓

– Single-shot Direct Linking

–

– –

Single-shot Direct Linking

– – –

– –

✓

✓ ✓ –

–

– –

✓

–

arXiv’25 arXiv’25 –

–

✓

Agentar-Scale-SQL N-rep arXiv’25

–

Pairwise Comparison

XiYan-SQL

Scoring and Ranking

–

✓

–

✓

–

✓

–

✓

–

✓

–

✓

–

Two-stage Direct Linking

–

✓

–

✓

–

✓

–

✓

–

✓

–

✓

–

✓ –

–

✓

✓ –

✓

–

COLING’25 COLING’25

–

COLING’25

DASFAA’25 –

Gen-SQL

PET-SQL

MAC-SQL

EMNLP’25

MCS-SQL

LinkAlign

–

✓

–

✓

–

✓

–

✓

–

Pairwise Comparison

Scoring and Ranking

–

✓

–

✓

–

✓

–

✓

–

✓

–

✓

–

✓

Pairwise Comparison

–

Single-shot Direct Linking

–

✓

–

✓

Two-stage Direct Linking

–

✓

–

Single-shot Direct Linking

–

✓

–

✓

–

Single-shot Direct Linking

–

✓

–

Single-shot Direct Linking

–

✓

–

ICML’25

– –

✓

Two-stage Direct Linking

–

✓

Alpha-SQL

– –

–

ICLR’25 –

✓

–

CHASE-SQL ICLR-W’25 – –

–

✓

–

ReFoRCE SIGMOD’25

✓

–

OpenSearch-SQL arXiv’24

✓

–

✓

–

RSL-SQL ACL-F’24 –

–

ICML-W’25

DEA-SQL ICDE’24

✓

–

CHESS-SQL

PURPLE VLDB’24

–

Two-stage Direct Linking

SuperSQL

–

✓

–

✓

–

✓

–

✓

–

Two-stage Direct Linking

–

Single-shot Direct Linking

–

✓

–

✓

–

arXiv’23

–

VLDB’24

–

✓

C3-SQL

NeurIPS’23

EMNLP-F’23

DAIL-SQL

DIN-SQL

ACT-SQL

Jiayan Lin, Yujia Liu, Zijin Hong, Zheng Yuan, Yilin Xiao, Hao Chen, Qinggang Zhang, Xiao Huang, and Feiran Huang

Example Retrieval. Shared setup. All three paradigms retrieve from the BIRD training split using a frozen bge-large-en-v1.5 encoder, with ℓ2 -normalized embeddings ranked by cosine similarity. Examples drawn from the target database are excluded to prevent leakage, and exact-duplicate questions are removed. The generator receives up to 𝑘=3 demonstrations, each rendered as the original question, its optional evidence annotation, and the gold SQL; no demonstration schema is included. Embeddings are precomputed once and cached. Paradigm

Retrieval signal

𝐸 1 Question Similarity E2 Masked Question Similarity

Raw question embedding. Question with schema-specific entities masked by the backbone before encoding; retrieval returns the original (unmasked) examples. Structural skeleton of a preliminary SQL draft, matched against skeletons of training gold SQL.

E3 SQL Skeleton Similarity

Shots 3 3 3

Auxiliary LLM calls. E2 issues one backbone call per query to produce the masked question; E3 issues one call to draft the preliminary SQL from which the skeleton is extracted. 𝐸 1 requires no auxiliary call. All auxiliary calls use the same backbone as the main generator, at temperature 0; on an empty or malformed response, the paradigm falls back to the unmasked question (E2 ) or to direct generation (E3 ). Skeleton extraction abstracts literals and non-keyword identifiers to placeholder tokens while preserving SQL keywords and operators. Trivial baseline. Zero-shot generation with no retrieved demonstrations. Schema Linking. Shared setup. Each paradigm receives the question, the full schema with column descriptions, and the optional evidence annotation, and returns a filtered schema for the generator; primary and foreign keys are always retained after filtering. Paradigms differ in who makes the linking decision. Malformed or empty responses fall back to the full schema, so no paradigm gains accuracy from its failure path. All auxiliary calls use the main backbone, so their cost is attributed to that backbone. Paradigm

Mechanism

𝑆 1 Retrieval-Based

Embeds question keywords against per-column text, boosts columns whose cells match a keyword, and keeps the top-scoring tables and columns. Drafts a SQL query over the full schema, then keeps the referenced tables and columns. Asks the backbone once to select the needed tables and columns, then validates the names against the schema. Selects relevant tables first, then selects columns only within those tables. A proposer selects a schema slice, a verifier judges its sufficiency, and an optional second proposer revises it.

S2 Pseudo-SQL Guided S3𝑎 Direct LLM, Single-shot S3𝑏 Direct LLM, Two-stage S4 Agentic

Calls 1 1 1 2 1–3

Auxiliary LLM calls. All linking calls run on the main backbone at temperature 0. 𝑆 1 uses a frozen bge-large-en-v1.5 encoder for its keyword–column matching. S4 issues a variable number of calls: a single proposer call when its proposal is empty or unparsable, two calls when the verifier accepts the slice, and a third only when the verifier reports a concrete deficiency. Trivial baseline. The full schema is passed to the generator unfiltered.

Figure 8: Controlled implementation of pre-processing modules: Example Retrieval and Schema Linking.

Are These Modules Worth Their Cost? A Paradigm-Level Accuracy-Cost Analysis of In-context Learning Text-to-SQL

Generation Strategy. Shared setup. All three paradigms receive the question, the schema (linked or full), and the optional evidence, and run on the main backbone at temperature 0 with a single sampled candidate—one generation call per query. Each parses the final SQL string from its own output and passes only that string downstream; when no SQL can be parsed, the query counts as incorrect. Paradigm

Mechanism

𝐺 1 Direct G2 CoT-Enhanced G3 Decomposition-Based

The backbone writes a single SQL query directly, returned in a fenced code block. The backbone emits a step-by-step reasoning trace and the final SQL in one structured response. The backbone decomposes the question into sub-questions and sub-queries and returns an assembled final SQL, all in one structured response.

Calls 1 1 1

Generation behavior. All three paradigms use a single backbone call; they differ only in what that call is asked to emit. In G2 , reasoning and SQL are produced together and is therefore charged as additional output tokens rather than obtained from a separate planning call. In G3 , the number of sub-questions is chosen by the model, and the final SQL is read directly from its output rather than assembled by executing sub-queries; G3 likewise adds no extra call. SQL is parsed from a structured field where available (G2 , G3 ) or from the fenced block (𝐺 1 ), with a shared fallback that extracts a bare SELECT/WITH statement. Trivial baseline. Direct generation (G1 ).

Figure 9: Controlled implementation of the generation stage: Generation Strategy.

Jiayan Lin, Yujia Liu, Zijin Hong, Zheng Yuan, Yilin Xiao, Hao Chen, Qinggang Zhang, Xiao Huang, and Feiran Huang

Candidate Selection. Shared setup. All selectors operate over the same pool of 𝑁 =5 candidates, sampled from the direct-generation prompt on the main backbone at temperature 0.8—the same pool used by the 𝑁 =5 reference baseline. Each selector returns one of the five candidates and never generates a new query. Paradigm

Mechanism

𝐶 1 Execution-Based Voting

Executes all five candidates, groups them by result equivalence, and takes the earliest candidate in the largest group. Shows the model all five candidates with their execution outcomes and asks it to pick the best. Runs a single-elimination tournament, comparing the running winner against each remaining candidate.

C2𝑎 Scoring and Ranking C2𝑏 Pairwise Comparison

LLM calls 0 1 𝑁 −1

Selection behavior. All selectors first execute the five candidates and run on the main backbone at temperature 0. They differ in cost: C1 issues no LLM call (execution only), C2𝑎 a single call independent of 𝑁 , and C2𝑏 𝑁 −1 calls (four at 𝑁 =5), the dominant cost driver among the selectors. Trivial baseline. Return the first candidate from the same 𝑁 =5 candidate pool. Refinement. Shared setup. Each paradigm receives the selected SQL, the question, the schema, and the optional evidence, and returns a final SQL that replaces the original for evaluation; if refinement yields no valid SQL, the original (or most recent valid revision) is kept. All refinement calls run on the main backbone at temperature 0. Paradigm

Mechanism

𝑅1 Execution-Feedback

Executes the SQL first, then asks the model to rewrite it only when execution errors or returns no rows. Asks the model to review the SQL against a generic error checklist, without executing it. A corrector revises the SQL using execution feedback, a verifier reviews the result, and the corrector may revise once more.

R2 Self-Correction R3 Agentic

LLM calls 0–1 1 1–3

Refinement behavior. The paradigms differ in when they fire, which is what separates their cost. R1 is failure gated: it runs one diagnostic execution and issues a model call only on an error or empty result, so its average call count per query is below one—the reason it adds accuracy at near-zero cost. R2 and R3 instead fire on every query: R2 issues exactly one review call with no execution feedback, while R3 issues two to three calls across its corrector and verifier roles. Each paradigm performs at most one revision round (R3 at most one additional corrector pass). Trivial baseline. The selected SQL is returned unchanged, with no refinement call.

Figure 10: Controlled implementation of post-processing modules: Candidate Selection and Refinement.

Are These Modules Worth Their Cost? A Paradigm-Level Accuracy-Cost Analysis of In-context Learning Text-to-SQL

Analyze & explore related research in Conceptio

Related documents