From Benchmarks to Production: A Text-to-SQL System for Complex Financial Data

From Benchmarks to Production: A Text-to-SQL System for Complex Financial Data Arijit Sehanobish* , Bruno Gomes Coelho* , Guillaume Michel, Sophia Zhi, Valerie Faucon-Morin, Kristen Howell Kensho Technologies Correspondence: arijit.sehanobish@kensho.com

arXiv:2610.03524v1 [cs.AI] 2 Oct 2026

Abstract General-purpose Text-to-SQL systems achieve strong performance on academic benchmarks like Spider and BIRD, where schemas are relatively shallow and column values are often human readable. In production financial databases, where concepts are stored as opaque integer keys rather than human-readable strings, these methods fall below 50%, as even simple queries require multiple joins and filter predicates reference opaque IDs. We present Financial LINking Text-to-SQL (FLINT), a domain-specialized Text-to-SQL system that closes this gap through three key components: (1) a lookup agent that dynamically resolves natural-language concepts to question-specific reference table constraints, (2) embeddingbased retrieval of structurally similar query templates from a compact, expert-authored bank, and (3) schema linking that prunes a large table schema to the relevant subset by traversing foreign-key chains, rather than relying on name similarity alone. We evaluate on two datasets totaling 359 questions over production financial schemas. FLINT outperforms various state-of-the-art baselines using the same LLM. The system is deployed in production as part of a financial data retrieval service.1

1

Introduction

Text-to-SQL, translating natural language questions into executable database queries, has emerged as a useful interface for non-technical users to access structured data. Recent systems have achieved 70-85% accuracy on academic benchmarks such as Spider (Yu et al., 2018) and BIRD (Li et al., 2023b) through advances in schema linking, multi-agent architectures, and prompt engineering. However, these results do not transfer to realworld database schemas in the financial sector. Questions about these databases (e.g., a company’s * Equal Contribution 1

https://kensho.com/solutions/adaptive-retrieval

quarterly revenue, the largest M&A deals in a sector) are straightforward to express in natural language, but difficult to translate into productionready SQL. Commercial financial databases optimize for storage efficiency, not human readability, so financial metrics, period types, and currencies are encoded as integer foreign keys across dozens of normalized reference tables. A single time-series question requires traversing a six-table join chain, and the filters that distinguish “annual revenue” from “quarterly net income” are opaque integers that cannot be inferred from column names alone (detailed examples in App. A.2 and A.3). This setting exposes a gap that academic benchmarks do not measure. In Spider (Yu et al., 2018) and BIRD (Li et al., 2023b), the column values are self-describing (e.g. country = ‘USA’), the joins are shallow (1-2 on average), and the schemas fit into a single prompt. We evaluate seven state-of-theart methods on two production financial datasets and find that none exceeds 50% accuracy, despite generating SQL with over 99% execution success. While the queries run, they often return the wrong data due to incorrect reference table filter values. We present Financial LINking Text-to-SQL (FLINT), a domain-specialized Text-to-SQL system that targets this through three components: (1) a lookup agent that dynamically resolves naturallanguage concepts to question-specific reference table constraints, (2) embedding-based retrieval over a compact, expert-authored query bank for structural templates, and (3) schema linking that prunes large schemas to relevant subsets, including the bridge tables needed for foreign-key traversal. This design decouples domain knowledge from model weights. When schemas evolve, new metrics are added, or coverage gaps are identified, domain experts update the configuration artifacts directly without retraining or ML engineering involvement. Our architecture addresses normalized schemas that are common across industries, and while we

focus on financial data, we validate the generalizability of this approach in additional scientific and industrial domains in App. G. To summarize, our contributions are: On normalized • Bottleneck diagnosis. production-style schemas, value grounding and not SQL generation is the dominant failure mode: 76–96% of errors are wrong reference-table filter IDs. • A lightweight value-grounding method. We introduce a model-independent configuration layer that targets this bottleneck. On four public benchmarks, it outperforms CHESS and ReFoRCE on three datasets at 5× lower generation cost. • Semantic evaluation. We introduce a threetier evaluation pipeline (AST canonicalization → column-aligned comparison → LLM judge) that distinguishes semantically equivalent queries from genuine errors.

2

Related Work

Prompt engineering approaches to Text-to-SQL (Gao et al., 2024; Pourreza and Rafiei, 2023, further discussion in App. I) have largely been superseded by multi-agent decomposition, where SQL generation is split across multiple specialized stages before the final query is produced. Recent systems address different failure modes: candidate diversity and selection (Talaei et al., 2024; Pourreza et al., 2025), interactive database exploration (Wang et al., 2025; Pham et al., 2026), and iterative refinement via execution feedback (Deng et al., 2025). While these strategies reduce structural SQL errors, none includes a dedicated mechanism for resolving opaque reference-table IDs. On normalized schemas in finance, this shared gap is the main source of error (see Section 4.3). Evaluation methodology and benchmarks remain active research concerns. Spider (Yu et al., 2018) compares SQL components structurally via exact set match, rejecting equivalent reformulations. In contrast, BIRD (Li et al., 2023b) looks only at query outputs, allowing formatting differences to be conflated with semantic errors. This is a limitation that SpotIt (Klopfenstein et al., 2026) quantifies through formal verification, hypothesizing that up to 20% of BIRD evaluation data may be incorrect because test databases coincidentally

agree on non-equivalent queries. Our benchmarks differ from Spider and BIRD in both content and evaluation: The schemas feature deep normalization with opaque IDs, mandatory multi-table joins for even simple queries, and implicit domain conventions absent from the natural language. Furthermore, we evaluate through a three-tier pipeline (AST canonicalization, output comparison, and LLM judge) designed to accommodate the many valid reformulations that arise on such schemas. We discuss additional related work in App. I.

3

System Architecture

FLINT decomposes natural language to SQL generation into a pipeline of specialized stages, each addressing a distinct sub-problem. Given a user question q, the system produces a SQL query ŝ through six stages: lookup, similar query retrieval, entity linking, schema linking, SQL generation, and execution with reflection. Stages without data dependency – lookup and similar query retrieval, followed by entity linking and schema linking – execute in parallel (Fig. 1). This parallelism yields median end-to-end latency of 10 - 20s (App. D.12). 3.1

Lookup Agent

Financial databases encode categorical information like industries, period types, and financial metrics as numeric IDs in reference tables. A question mentioning “quarterly EBITDA” requires knowing which integer ID maps to “Quarterly” and which maps to “EBITDA”. This cannot be inferred from the schema, which shows only that these are integer foreign keys (Fig 4). The Lookup Agent resolves these references in two sub-steps: 1. Table selection: An LLM determines which reference tables are relevant to the question. A question about “Canadian companies” triggers a geography table lookup; “revenue” triggers the financial metrics table. 2. Constraint generation: The agent executes SELECT queries against the identified production reference table(s), presents retrieved rows to the LLM, and asks it to produce the appropriate SQL filters. On Financials, metric resolution is a two-stage funnel: the full 11,412-item metrics table is first reduced to an SME-curated catalog of ∼2,500 commonly-used items (the remaining ∼9,000 are rarely queried, and the curation is itself

evaluation set are strictly disjoint. Retrieved examples serve two roles: structural templates for schema linking (showing correct join patterns) and SQL exemplars for generation (demonstrating syntax and filter placement).

Question q

Similar Query Retrieval

Lookup Agent

Entity Linking

Schema Linking

SQL Generation

success

Results

Execute SQL

error

Reflection

Figure 1: FLINT pipeline. Stages arranged horizontally execute in parallel. Dashed arrows show the error-driven reflection loop.

domain knowledge), and a per-question fuzzymatching step then narrows this to the ∼50– 100 candidates from which the LLM selects. The Transactions reference tables are small and passed whole. On our evaluation set, this pre-filtering maintains 98% recall (measured against gold query). The output is a set of lookup constraints: SQL WHEREclause fragments passed to subsequent stages. 3.2

Similar Query Retrieval

FLINT maintains a query bank of expert-authored examples, each containing a question, reasoning traces, and reference SQL. At inference, the k=3 most similar examples are retrieved via cosine similarity over a vector store. Each example is hand-crafted by a domain expert to cover a diverse set of distinct query patterns (e.g., single-entity lookup,2 multi-period comparison, cross-currency ranking). The query bank and 2 Query bank examples use a canonical entity (a fixed reference company) to ensure similarity captures the type of data retrieval rather than surface-level entity overlap.

3.3

Entity Linking

Entity linking resolves natural-language mentions (company names, ticker symbols) to the opaque integer keys used in the database. Without this step, the model has no path from a mention like “Nvidia” to the correct numeric identifier, as the core data tables contain no human-readable name column, and fuzzy string matching against 20 million+ entities is unreliable at production scale. FLINT delegates resolution to an external entity disambiguation service, Kensho Link,3 that handles name variations and identifier-type disambiguation, returning canonical integer IDs. Resolved IDs are injected as concrete WHERE-clause constraints into the generation prompt. 3.4

Schema Linking

Schema linking determines which tables, columns, join conditions, and filter predicates are required for the query. Given the question, reference table constraints, lookup IDs, and retrieved examples, the LLM selects the relevant subset of the schema. This is challenging because the answer rarely lives in a single table and reaching it requires identifying intermediate tables that serve only as join bridges. For example, retrieving a stock price for a company requires resolving the company’s security, then its trading item, before reaching the price table: Company → Security → TradingItem → PriceEquity. The prompt includes: (1) the full database schema with column descriptions and documented join paths, (2) lookup constraints from the Lookup Agent (resolving which integer IDs to filter on), (3) domain-specific instructions encoding implicit conventions (e.g., default temporal granularity, which of several price columns to use, required flag filters for primary listings), and (4) retrieved examples with their reasoning traces. The LLM outputs a structured representation consisting of selected columns, join conditions, and filters. This decouples what to query from how to write SQL: Schema linking reasons about table selection while generation handles syntax. Since 3

https://kensho.com/solutions/core-ai-capabilities#link

schema linking sees the full schema, this design suits our target of a fixed, curated schema. 3.5

SQL Generation

Given schema links, entity IDs, and SQL exemplars, the LLM generates the final query. Domain constraints are enforced via the prompt: Use only specified tables and columns, apply entity IDs as concrete filters, include contextual columns (company name, currency) for interpretability, and follow the target SQL dialect. Before execution, a safety validator ensures the query only accesses allowed schemas and tables, preventing unintended data exposure. 3.6

Execution and Reflection

The query is executed against the database. On execution success, results are returned directly. On failure, the system enters a reflection loop: The LLM receives the original question, failed SQL, and database error, then regenerates with targeted corrections. This reflection step is particularly useful for complex schemas, where execution errors often stem from localized mistakes, such as missing join conditions, incorrect aliases, or column typos, which become diagnosable once the error message is available. While we allow only one reflection attempt to stay within production latency constraints, this is configurable for use cases where accuracy is prioritized over response time. 3.7

Dataset Configuration

FLINT adapts to new datasets without fine-tuning or retraining. Configuration requires four artifacts: 1. A schema CSV mapping tables and columns to descriptions and join-path documentation. 2. A list of reference tables for the Lookup Agent (identifying tables with categorical IDs). 3. A query bank of 15–30 examples covering major query patterns. 4. Domain-specific instruction prompts for schema linking and generation (encoding implicit conventions). Because these artifacts are structured domain knowledge, they are authored by subject-matter experts rather than ML engineers. In our experience a moderate-complexity dataset takes roughly one person-week to configure, and a complex one

(many reference tables, several implicit conventions) takes two to three weeks including iterative refinement; adding a new example for a novel pattern is ∼30 minutes (App. B). The pipeline is general: Adapting FLINT to a new financial dataset requires only new configuration artifacts, not implementation changes. FLINT’s deployment details, including caching, fallback strategies, and latency constraints, is presented in App. H.

4

Experiments

We first describe our proprietary datasets (Sec. 4.1), our multi-stage evaluation protocol (Sec. 4.2), compare our results with various well known multiagent systems (Sec. 4.3), and detailed ablation studies (Sec. 4.4), and error analysis (Sec. 4.5). 4.1

Datasets

We evaluate FLINT on two datasets constructed over a large-scale financial database with deep schema normalization, summarized in Table 1. The Financials dataset covers company intelligence, market data, financial metrics, and securities across global public companies. The schema consists of 24 tables (8 core data tables, 16 reference tables). Financial values are not stored with humanreadable labels but metric names, period types, unit types, and currencies are each encoded as integer foreign keys into separate reference tables. A typical query requires joining 6 tables; complex screening queries require up to 13 joins. The dataset contains 221 natural-language questions authored by domain experts, covering company fundamentals (44%), market data (14%), market capitalization (10%), screening/ranking (8%), industry classification (7%), and other mixed queries (17%). The Transactions dataset covers M&A activity, funding rounds, stock buybacks, and corporate relationships. The schema consists of 18 tables (6 core, 12 reference/lookup) capturing deal types, advisor roles, company relationships, and transaction statuses. Queries require joining 5 tables on average to resolve entity relationships and transaction details. The dataset contains 138 natural-language questions authored by domain experts, covering acquisitions (50%), stock buybacks (14%), general transaction screening (14%), bankruptcies (8%), spin-offs and divestitures (6%), investments and funding (4%), and other deal types (4%). The 359 evaluation questions are drawn from a pool of production queries logged over 6 months,

Table 1: Schema and benchmark statistics for both datasets. Reference tables store lookup values (metrics, period types, currencies); core tables store financial data and transactions.

Parse gold & generated SQL

AST Comparison LLM Judge mismatch SQL Output Comparison mismatch sqlglot, tree edit distance column alignment, set equality semantic equivalence

Property

Financials

Transactions

Total tables Reference / lookup Core data

24 16 8

18 12 6

Avg. joins per query Max joins (single query) Queries requiring CTEs

6.0 13 40

5.2 13 17

Evaluation questions

221

138

deduplicated and filtered to ensure coverage across query categories, table usage, and difficulty levels. Each question is paired with a gold SQL query written by database domain experts.4 The gold queries enforce constraints central to correctness on normalized schemas: explicit joins, required flag filters for selecting primary listings, currency normalization for cross-country comparisons, and correct temporal-granularity constraints. These are precisely the constraints that general-purpose Text-to-SQL systems fail to satisfy (Sec. 4.3). While we focus on the financial domain, the constraints we address are present in other domains. We validate the generalizability of our system on two external benchmarks spanning various scientific and industrial domains in App. G. The query bank used for few-shot retrieval (27 for Financials, 18 for Transactions) (Section 3.2) is disjoint from the evaluation set: No evaluation question appears in, was derived from, or shares an entity with any query bank example. To quantify residual similarity, we measure the normalized treeedit distance (TED) from each evaluation query to its nearest query-bank neighbor over sqlglot ASTs (overlap = 1−TED). Content-aware overlap is small with no near-duplicate on Transactions, so the query bank supplies structural patterns and not answers (App. F). 4.2

Evaluation

SQL queries that are syntactically different can produce identical results, while structurally similar queries can return different data due to subtle filter differences. We use a three-tier evaluation pipeline: 1. Execution: We execute both gold and generated queries against the production database; 4 This may bias the benchmark toward questions with known SQL solutions, which we consider acceptable for measuring system coverage of production use cases.

match

match

verdict

Score

Figure 2: Our multi-stage evaluation pipeline: The generated queries that execute successfully are first matched via AST trees with the gold query. For the unmatched queries we pass them through heuristics and eventually to LLM-as-a-judge.

execution failures score 0 immediately. 2. Heuristics: We parse both queries into abstract syntax trees using sqlglot, apply canonicalizing transformations (alias normalization, date function standardization, JOIN reordering), and compute tree edit distance. Queries that differ only in aliasing, column order, or semantically equivalent reformulations (e.g., CTE vs. subquery) are matched. For queries that are not matched via AST, gold columns are checked against the generated columns via case-insensitive name alignment, and values are compared with order-invariant set equality. This heuristic resolves almost 60% of cases at negligible cost. 3. LLM Judge: We prompt an LLM to determine semantic equivalence by comparing result tables with reference to the original question. The LLM judge decides only ∼40% of comparisons (the rest are settled by heuristics or execution), and on a validated sample it agrees with human judgments ≥94% of the time, bounding a ±3-point worst case on the reported accuracies. We report accuracy as the percentage of questions whose gold and generated SQL are marked equivalent in step 2 or 3 (additional details in App. C). 4.3

Comparison with Existing Methods

We compare against seven Text-to-SQL systems spanning multi-agent architectures (CHESS (Talaei et al., 2024), Chase-SQL (Pourreza et al., 2025), MAC-SQL (Wang et al., 2025), FlexSQL (Pham et al., 2026), ReFoRCE (Deng et al., 2025)) and prompt-engineering approaches (DAIL-SQL (Gao et al., 2024), DIN-SQL (Pourreza and Rafiei, 2023)) (Tab. 2). All use Claude 4.5 Opus (Anthropic, 2025).

Table 2: Comparison with existing Text-to-SQL methods on normalized financial schemas. All methods use Claude 4.5 Opus and share base infrastructure (App. D.1); DAIL-SQL, DIN-SQL, and FlexSQL additionally receive shared pipeline components (App. D.10), making their comparison conservative; execution success is ≥99% for all.

Method

Approach

Fin. Trans. Accuracy (%)

FLINT

Domain-specialized agents

68.8

65.2

Self-consistency (k=1) Self-consistency (k=10) Multi-path + tournament Agentic decomposition Plan + schema explore Iterative refinement

27.1 35.6 16.3 33.3 43.4 46.6

42.0 49.6 21.0 37.3 40.6 47.1

Few-shot selection Decompose + classify

31.7 33.5

39.9 36.5

12.7 38.0

15.9 42.8

M ULTI - AGENT / MULTI - CANDIDATE CHESS (Talaei et al., 2024) CHESS (Talaei et al., 2024) Chase-SQL (Pourreza et al., 2025) MAC-SQL (Wang et al., 2025) FlexSQL (Pham et al., 2026) ReFoRCE (Deng et al., 2025) P ROMPT ENGINEERING DAIL-SQL (Gao et al., 2024) DIN-SQL (Pourreza and Rafiei, 2023)

D IRECT PROMPTING ( ENTITY LINKING ONLY ) Zero-shot Few-shot (k=3)

All methods produce valid SQL (≥99% execution success) but fail semantically: The dominant error is syntactically correct queries with wrong filter values. No baseline exceeds 50% accuracy, despite achieving substantially higher performance on Spider/BIRD where column values are often semantically informative (e.g. country = ‘USA’). On our schemas, mapping from natural language (“annual revenue”) to opaque integer IDs requires querying reference tables that no baseline provides. CHESS k=10 gains 8.5 points over k=1 on Financials via sampling diversity (occasionally hitting correct IDs), but self-consistency cannot fix errors shared across all candidates. Among baseline failures, 64–96% contain incorrect IDs (App. D). These gaps are well outside sampling noise: Wilson 95% intervals on FLINT are 68.8% [62.4, 74.5] (Financials) and 65.2% [56.9, 72.6] (Transactions), and the margin over the strongest baseline exceeds them by a wide margin; the judge’s ±4-point worst case cannot close a gap this size. Moreover, FLINT is the fastest system evaluated, completing queries in 10–20s median latency, faster than all seven baselines (Fig. 5). Generalization to other domains. Moreover, to test domain generality, we evaluate FLINT on two external benchmarks spanning medical, scientific, and public-sector domains; with only minor configuration changes (schema documentation, query bank, and domain rules), FLINT outperforms both

CHESS and ReFoRCE while remaining almost 5× faster. App. G provides further discussion, additional results on SM3 (Sivasubramaniam et al., 2024) and comparisons with few-shot prompting. FLINT (ours)

CHESS

ReFoRCE

90 85 80

Accuracy (%)

Claude 4.5 Opus Claude 4.5 Opus

75

CORDIS OncoMX SDSS

70 65 60 55 50 2

5

10

20

50

100

Median generation time (s)

Figure 3: Accuracy vs. generation time on ScienceBenchmark (CORDIS, OncoMX, SDSS), under one harness and LLM. Top-left is best—faster and more accurate. FLINT is the most accurate on every database while generating SQL 5× faster than the multi-agent systems: CHESS issues k=10 candidate generations per question yet trails FLINT’s single pass, and ReFoRCE’s iterative refinement cannot recover semantically wrong filters that still execute. Generation time excludes database execution.

4.4

Ablation Study

We isolate each component’s contribution by disabling it while keeping the rest intact (Tab. 3). For w/o Lookup, the prompt includes static mappings for the 100 most common Financials IDs and 50 most common Transactions IDs, covering

Table 3: Ablation study (strict accuracy, %). Each row dis-

Table 4: Error taxonomy for FLINT’s 117 incorrect predic-

ables one component while keeping the rest of the pipeline intact. Rows sorted by average degradation.

tions. All failures produce executable SQL. Error Category

Financials ∆

Transactions

Acc.

FLINT (full system)

68.8

—

65.2

—

54.8 57.9 65.2 67.9

−14.0 −10.9 −3.6 −0.9

33.3 40.6 50.7 55.1

−31.9 −24.6 −14.5 −10.1

w/o Lookup Agent w/o Few-Shot Retrieval w/o Schema Linking w/o Reflection

Acc.

Correct tables & columns Wrong filter / aggregation Correct tables, wrong columns Partial match (≥50%) Low match (<50%) Wrong table selection Missing / extra tables

∆

Configuration

Total

about 78% of queries. w/o Schema Linking, the full schema is passed, w/o Retrieval, no exemplar queries are provided, and w/o Reflection, the first SQL is submitted without retry (details in App. E). The lookup agent is the most critical component: removing it drops accuracy by 14 points on Financials and 32 points on Transactions. Even though static mappings cover most of the queries, it cannot replace the question-specific disambiguation required to select the correct identifier. Retrieval is the second-largest contributor (11/25-point drop) despite relying on only 27 examples, indicating that diversity matters more than scale. Schema linking is especially important for Transactions (15-point drop versus 4 on Financials), where it resolves ambiguity among semantically similar columns. Reflection helps only when execution errors occur, yielding little benefit on Financials but recovering 15 of 22 failures on Transactions. 4.5

Error Analysis

FLINT’s 117 incorrect predictions fall into three main categories (Tab 4). Column selection is the primary failure mode, accounting for 71 errors. Among partial-match cases, some return the correct values with an extra or missing non-essential column (presentation-only issues), while genuine mistakes arise from metric ambiguity (e.g., mapping “stock price” to adjusted close, last close, or OHLC) or from selecting an identifier column rather than its corresponding value column. 30 failures occur after the correct tables and columns have been selected, due to errors in lookup constraints. On Financials, all 15 such cases are caused by incorrect lookup resolution despite otherwise correct schema grounding. Table selection errors are uncommon (4.5% of queries), and entity linking contributes 5 failures on Financials and 11 on Transactions. Table 4 counts failures by strict exact-match, which scores a query wrong whenever its result ta-

Fin.

Trans.

Tot.

15

15

30

35 11

15 10

50 21

8

8

16

69

48

117

ble differs from the gold table. A separate manual audit finds that many of the ≥50% partial-match failures return the correct values but add or omit a non-essential column. Under this interpretation, effective accuracy rises to ∼85% (Financials) / ∼76% (Transactions), as these cases are treated as correct despite being classified as substantive errors by the strict taxonomy (App. D.11). The remainder is concentrated in value grounding, metric/column disambiguation and lookup-constraint resolution. All failure modes remain inspectable in deployment, where the analyst sees the generated SQL, the resolved IDs, and the intermediate steps.

5

Conclusion

We presented Financial LINking Text-to-SQL (FLINT), a Text-to-SQL system designed for production financial databases with deep schema, opaque integer keys as values, and questions that carry implicit domain constraints absent from the natural language. FLINT combines schema linking, example retrieval, entity resolution, domainspecific prompt rules, and execution-based reflection into a single-pass pipeline that runs in 10–20 seconds per query. On two proprietary financial benchmarks, FLINT outperforms the seven academic baselines. Across four additional scientific and medical domains, the same architecture generalizes with only minimal changes. Our results suggest that the gap between academic Text-to-SQL benchmarks and production deployment is not primarily about SQL generation capability, but about the surrounding domain knowledge: knowing which tables to join, what values mean, and the intent of user questions. FLINT’s architecture offers one reusable pattern for closing that gap.

Limitations FLINT’s domain rules are manually authored by subject matter experts familiar with each schema’s conventions. While effective, this introduces a perdomain configuration cost that does not scale automatically to new databases without human effort. Similarly, the retrieval component requires a curated query bank. On a new schema with no historical queries, this component is unavailable and accuracy degrades. One component of our evaluation pipeline uses an LLM as judge for semantic equivalence; while we validate it against human annotations on a held-out subset, its agreement is imperfect and may introduce noise into reported accuracy. Finally, FLINT operates in a single-turn setting: It generates one SQL query per natural language question and does not support conversational refinement such as follow-up filters or corrections that reference prior context.

References Anthropic. 2025. Claude opus 4.5. Blog post. Minghang Deng, Ashwin Ramachandran, Canwen Xu, Lanxiang Hu, Zhewei Yao, Anupam Datta, and Hao Zhang. 2025. Reforce: A text-to-sql agent with selfrefinement, consensus enforcement, and column exploration. arXiv preprint arXiv:2502.00675. Xuemei Dong, Chao Zhang, Yuhang Ge, Yuren Mao, Yunjun Gao, lu Chen, Jinshu Lin, and Dongfang Lou. 2023. C3: Zero-shot text-to-sql with chatgpt. Preprint, arXiv:2307.07306. Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. 2024. Text-to-sql empowered by large language models: A benchmark evaluation. Proc. VLDB Endow., 17(5):1132–1145. Rocky Klopfenstein, YANG HE, Andrew Tremante, Yuepeng Wang, Nina Narodytska, and Haoze Wu. 2026. Spotit: Evaluating text-to-SQL evaluation with formal verification. In The Fourteenth International Conference on Learning Representations. Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen. 2023a. Resdsql: decoupling schema linking and skeleton parsing for text-to-sql. In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence, AAAI’23/IAAI’23/EAAI’23. AAAI Press. Jinyang Li, Binyuan Hui, GE QU, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying

Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2023b. Can LLM already serve as a database interface? a BIg bench for large-scale database grounded text-to-SQLs. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track. Quang Hieu Pham, Yang He, Ping Nie, Canwen Xu, Davood Rafiei, Yuepeng Wang, Xi Ye, and Jocelyn Qiaochu Chen. 2026. Flexsql: Flexible exploration and execution make better text-to-sql agents. Preprint, arXiv:2605.02815. Mohammadreza Pourreza, Hailong Li, Ruoxi Sun, Yeounoh Chung, Shayan Talaei, Gaurav Tarlok Kakkar, Yu Gan, Amin Saberi, Fatma Ozcan, and Sercan Arik. 2025. Chase-sql: Multi-path reasoning and preference optimized candidate selection in text-to-sql. In International Conference on Learning Representations, volume 2025, pages 60385–60415. Mohammadreza Pourreza and Davood Rafiei. 2023. DIN-SQL: Decomposed in-context learning of textto-SQL with self-correction. In Thirty-seventh Conference on Neural Information Processing Systems. Sithursan Sivasubramaniam, Cedric Osei-Akoto, Yi Zhang, Kurt Stockinger, and Jonathan Fuerst. 2024. SM3-text-to-query: Synthetic multi-model medical text-to-query benchmark. In The Thirtyeight Conference on Neural Information Processing Systems Datasets and Benchmarks Track. Shayan Talaei, Mohammadreza Pourreza, Yu-Chen Chang, Azalia Mirhoseini, and Amin Saberi. 2024. Chess: Contextual harnessing for efficient sql synthesis. arXiv preprint arXiv:2405.16755. Bing Wang, Changyu Ren, Jian Yang, Xinnian Liang, Jiaqi Bai, LinZheng Chai, Zhao Yan, Qian-Wen Zhang, Di Yin, Xing Sun, and Zhoujun Li. 2025. MACSQL: A multi-agent collaborative framework for textto-SQL. In Proceedings of the 31st International Conference on Computational Linguistics, pages 540– 557, Abu Dhabi, UAE. Association for Computational Linguistics. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, and 1 others. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 3911–3921. Yi Zhang, Jan Deriu, George Katsogiannis-Meimarakis, Catherine Kosten, Georgia Koutrika, and Kurt Stockinger. 2023. Sciencebenchmark: A complex real-world benchmark for evaluating natural language to sql systems. Proc. VLDB Endow., 17(4):685–698.

A

Dataset Examples

We present representative examples from different difficulty levels to illustrate the structural complexity of our benchmarks. Table names are shortened for readability (e.g., Company for the full internal table name). A.1

Difficulty levels

We stratify questions by the structural complexity of the gold SQL (join count and CTE presence). Join count is an imperfect proxy for difficulty since a 3-join query requiring implicit domain knowledge can be harder than a routine 7-join traversal. However it correlates with the number of opportunities for error accumulation and is objectively measurable. We use three levels: • Simple (1–3 joins, no CTE): Entity lookups, single-table retrievals, basic filtering. • Moderate (4–6 joins, no CTE): Standard financial data retrieval requiring the full join chain through period and metric tables. • Complex (7+ joins or CTE): Multi-entity comparisons, cross-currency rankings, temporal aggregations. Tab. 5 shows the distribution. Table 5: Question difficulty distribution by structural complexity of the gold SQL. Financials Difficulty Simple (1–3 joins) Moderate (4–6 joins) Complex (7+ joins / CTE)

A.2

Transactions

n

%

n

%

22 142 57

10 64 26

38 66 34

28 48 25

Financials Dataset

Simple (1–3 joins). “Where is Town Pump based out of?” A geographic lookup requiring two joins to resolve state and country names from foreign key IDs. SELECT Company . companyName , State . state , Country . country FROM Company JOIN State ON Company . stateId = State . stateId JOIN Country ON Company . countryId = Country . countryId WHERE Company . companyId = 4222311

2 joins. The companyId is resolved via entity linking from “Town Pump.” Moderate (4–6 joins). “Nvidia revenue from 2020 to 2025” A standard financial data retrieval requiring the full six-table join chain through period types, metrics, units, and currencies. SELECT Company . companyName , FinancialData . dataItemValue AS revenue , DataItem . dataItemName , UnitType . unitTypeName AS unit , FinPeriod . fiscalYear , Currency . currencyName FROM FinancialData JOIN DataItem ON DataItem . dataItemId = FinancialData . dataItemId JOIN FinPeriod ON FinPeriod . financialPeriodId = FinancialData . financialPeriodId JOIN Company ON Company . companyId = FinPeriod . companyId JOIN PeriodType ON FinPeriod . periodTypeId = PeriodType . periodTypeId JOIN UnitType ON FinancialData . unitTypeId = UnitType . unitTypeId JOIN Currency ON FinPeriod . currencyId = Currency . currencyId WHERE Company . companyId = 32307 AND FinancialData . dataItemId = 28 AND FinPeriod . fiscalYear BETWEEN 2020 AND 2025 AND PeriodType . periodTypeId = 1 ORDER BY FinPeriod . fiscalYear DESC

6 joins. The dataItemId (28 = Total Revenue) and periodTypeId (1 = Annual) must be resolved from lookup tables. This join chain is the “backbone” of most Financials queries. Complex (7+ joins / CTE). “2024 annual revenue for Alibaba” Identical intent to the previous example, but for a non-US company whose financials are reported in CNY. A currency conversion CTE is required to normalize to USD. WITH CurrencyConversion AS ( SELECT ExchangeRate . priceDate , ExchangeRate . currencyId , Currency . isoCode AS isoCodeSource , ER2 . currencyId AS currencyIdTarget , C2 . isoCode AS isoCodeTarget , ER2 . priceClose / ExchangeRate . priceClose AS exchangeRate FROM ExchangeRate JOIN Currency ON ExchangeRate . currencyId = Currency . currencyId JOIN ExchangeRate ER2

ON ExchangeRate . priceDate = ER2 . priceDate JOIN Currency C2 ON ER2 . currencyId = C2 . currencyId WHERE ExchangeRate . latestSnapFlag = 1 AND ER2 . latestSnapFlag = 1 AND Currency . majorCurrencyFlag = 1 AND C2 . majorCurrencyFlag = 1

) SELECT Company . companyName , FinancialData . dataItemValue , DataItem . dataItemName , UnitType . unitTypeName AS unit , FinPeriod . fiscalYear , Currency . currencyName , FinancialData . dataItemValue * CurrencyConversion . exchangeRate AS revenueUSD FROM FinancialData JOIN DataItem ON DataItem . dataItemId = FinancialData . dataItemId JOIN FinPeriod ON FinPeriod . financialPeriodId = FinancialData . financialPeriodId JOIN Company ON Company . companyId = FinPeriod . companyId JOIN PeriodType ON FinPeriod . periodTypeId = PeriodType . periodTypeId JOIN UnitType ON FinancialData . unitTypeId = UnitType . unitTypeId JOIN Currency ON FinPeriod . currencyId = Currency . currencyId JOIN CurrencyConversion ON CurrencyConversion . priceDate = FinPeriod . periodEndDate AND CurrencyConversion . currencyId = FinPeriod . currencyId WHERE Company . companyId = 42083601 AND FinancialData . dataItemId = 28 AND FinPeriod . fiscalYear = 2024 AND PeriodType . periodTypeId = 1 AND CurrencyConversion . isoCodeTarget = ' USD '

10 joins (3 in CTE + 7 in main query). The CTE computes cross-rates between all major currency pairs using a triangulation through USDdenominated exchange rates. This pattern recurs in any query comparing entities across currencies. A.3

Transactions Dataset

Simple (1–3 joins). “How many IPOs were cancelled in 2020?” An aggregation query filtering on transaction type and status, both resolved via reference table IDs. SELECT COUNT ( tr . transactionId ) AS cancelledIPOs FROM Transaction tr JOIN TransactionType tt ON tr . transactionIdTypeId = tt . transactionIdTypeId JOIN TransactionStatus ts ON tr . statusId = ts . statusId WHERE tr . transactionIdTypeId = 6 AND tr . statusId = 3

AND tr . announcedYear = 2020

2 joins. The transactionIdTypeId (6 = IPO) and statusId (3 = Cancelled) are resolved from reference tables. Moderate (4–6 joins). “When did Amazon acquire Whole Foods?” A specific acquisition lookup requiring traversal of the buyer–target relationship chain through three intermediate tables. SELECT target . companyName AS " Target " , buyer . companyName AS " Buyer " , tr . announcedDay , tr . announcedMonth , tr . announcedYear FROM Transaction tr JOIN TransactionType trtype ON tr . transactionIdTypeId = trtype . transactionIdTypeId JOIN Company target ON tr . companyId = target . companyId JOIN TransactionCompanyRel cr ON cr . transactionId = tr . transactionId JOIN TransactionRelType crt ON crt . transactionToCompRelTypeId = cr . transactionToCompRelTypeId JOIN CompanyRel crel ON crel . companyRelId = cr . companyRelId JOIN Company buyer ON buyer . companyId = crel . companyId WHERE target . companyId = 36602 AND buyer . companyId = 18749 AND trtype . transactionIdTypeId = 2 AND crt . transactionToCompRelTypeId = 1

6 joins. The four-table chain Transaction → TransactionCompanyRel → CompanyRel → Company is the standard path to resolve buyer/seller relationships and recurs in most M&A queries. Complex (7+ joins / CTE). “10 largest M&A deals in the financial sector in 2022” A ranking query combining sector filtering, currency normalization, and transaction-size ordering. WITH CurrencyConversion AS ( SELECT ExchangeRate . priceDate , ExchangeRate . currencyId , Currency . isoCode AS isoCodeSource , ER2 . currencyId AS currencyIdTarget , C2 . isoCode AS isoCodeTarget , ER2 . priceClose / ExchangeRate . priceClose AS exchangeRate FROM ExchangeRate JOIN Currency ON ExchangeRate . currencyId = Currency . currencyId JOIN ExchangeRate ER2 ON ExchangeRate . priceDate = ER2 . priceDate

JOIN Currency C2 ON ER2 . currencyId = C2 . currencyId WHERE ExchangeRate . latestSnapFlag = 1 AND ER2 . latestSnapFlag = 1 AND C2 . majorCurrencyFlag = 1

) SELECT tr . transactionId , Company . companyName , tr . announcedYear , TransactionType . transactionIdTypeName , Industry . simpleIndustryDescription , tr . transactionSize * exrate . exchangeRate AS transactionSizeUSD FROM Transaction tr JOIN TransactionType ON tr . transactionIdTypeId = TransactionType . transactionIdTypeId JOIN Company ON tr . companyId = Company . companyId JOIN Industry ON Company . simpleIndustryId = Industry . simpleIndustryId JOIN CurrencyConversion exrate ON tr . currencyId = exrate . currencyId AND DATE_FROM_PARTS ( tr . announcedYear , tr . announcedMonth , tr . announcedDay ) = exrate . priceDate WHERE Industry . simpleIndustryId IN (46 ,48 ,49 ,50 ,51) AND TransactionType . transactionIdTypeId = 2 AND tr . announcedYear = 2022 AND tr . transactionSize > 0 AND exrate . isoCodeTarget = ' USD ' ORDER BY transactionSizeUSD DESC LIMIT 10

7 joins (3 in CTE + 4 in main query). Combines the currency conversion CTE with industry filtering. The IN (46,48,49,50,51) clause reflects the multiple sub-industry IDs that map to “Financial Sector”, another reference table resolution challenge. A.4

Key Observations

These examples illustrate three properties that make our benchmarks challenging for generalpurpose Text-to-SQL systems: 1. Deep normalization: Even simple questions require resolving opaque integer IDs (e.g., dataItemId = 28 for revenue, periodTypeId = 1 for annual) from lookup tables. Without a dedicated lookup mechanism, models cannot produce correct filter predicates (Fig. 4). 2. Long join chains: The standard financial data

Schema declaration (visible to the model) ciqFinancialData.dataItemId INTEGER REFERENCES ciqDataItem(dataItemId) ciqFinancialData.periodTypeId INTEGER REFERENCES ciqPeriodType(periodTypeId) Question: “What was the quarterly EBITDA?” ⇓ L OOKUP AGENT QUERIES REFERENCE TABLES ⇓ Reference tables 1 2 3

ciqPeriodType ciqDataItem Annual 28 Total Revenue Quarterly 42 EBITDA LTM 105 Net Income

⇓ Resolved: periodTypeId=2 AND dataItemId=42

Figure 4: Opaque identifiers in Financials. The schema exposes only integer foreign keys. As a result, resolving “quarterly EBITDA” to concrete filter values requires querying reference tables, a step the Lookup Agent performs automatically.

retrieval pattern requires 6 joins as a baseline. This is the “moderate” difficulty, i.e. the minimum join path for accessing any time-series financial metric. 3. Currency normalization: Any cross-country comparison requires a CTE that triangulates exchange rates, adding 3 joins and substantial SQL complexity. This pattern cannot be discovered through schema inspection alone. It requires domain knowledge about how the exchange rate table encodes USD-denominated rates. A.5

Benchmark Size

Our benchmark (359 questions) is smaller than Spider (1,034 dev) or BIRD (1,534 dev). We argue this is sufficient for two reasons. First, both datasets share a single fixed schema, eliminating the cross-database variance that inflates sample size requirements in multi-schema benchmarks. Second, a 95% Clopper-Pearson confidence interval at 77% accuracy with n=221 is ±5.5 points, narrow enough to distinguish FLINT from all baselines at p < 0.001. The benchmark is not intended to estimate fine-grained performance differences between systems within a few points of each other; rather, it is designed to measure whether domain-specialized infrastructure closes the gap that general-purpose methods cannot.

A.6

Comparison with Existing Text-to-SQL Benchmarks

Table 6 contrasts our benchmarks with Spider (Yu et al., 2018) and BIRD (Li et al., 2023b). Table 6: Comparison with Spider and BIRD. Key differentiators: deep normalization with opaque IDs, mandatory multitable joins for even simple queries, and production-deployed schemas. Property

Spider BIRD

Ours

Avg. joins per query Max joins (single query) Reference/lookup tables Opaque integer IDs Currency conversion Self-describing values Schema size (tables) Production deployment

1.2 4 0 × × ✓ 4–8 ×

5.6 13 12–16 ✓ ✓ × 18–24 ✓

1.8 6 0–2 × × ✓ 5–15 ×

Three properties distinguish our benchmarks: Opaque schemas. Spider and BIRD filter on human-readable values (country = ‘USA’). Our schemas encode concepts as integer foreign keys (dataItemId = 28 for “Total Revenue”), requiring a lookup step that no existing benchmark demands. Structural depth as baseline. On Spider, 78% of queries use 0 to 1 joins. On our benchmarks, most of our queries use 4 – 6 joins. The standard path traverses Company → FinPeriod → FinancialData → DataItem → PeriodType → UnitType → Currency. BIRD increases schema scale modestly but not join depth, as about 84% of the queries have at most 1 join. Implicit domain conventions. Questions carry constraints absent from the natural language. For example, “stock price” implies priceClose; “revenue” implies annual (periodTypeId = 1); “market cap” requires primaryFlag = 1; and crosscountry comparisons require currency normalization. Spider and BIRD questions are self-contained, while ours require domain knowledge to disambiguate.

B

Configuration Cost

The bulk of configuration effort is authoring the query bank: relevant questions, their gold SQL, and a reasoning trace. Each takes longer than a routine evaluation question (which averages ∼30 minutes including 2–3 review rounds) because of the

chain-of-thought walkthrough, roughly 12 personhours for Transactions and 18 person-hours for Financials. The schema CSV is usually straightforward, since production databases already carry metadata or customer-facing documentation. Instruction prompts and example traces add ∼4–8 person-hours for a complex schema. All in, a moderate-complexity dataset takes roughly a person-week, while a complex one (24 tables, 16 reference tables, multiple implicit conventions) two to three weeks including iterative refinement as production traffic surfaces gaps. Once configured, adding an example for a novel pattern is ∼30 minutes. Sensitivity is asymmetric across both artifacts and datasets. Domain rules are a prerequisite. The rules alone can beat various multi-agent systems. Returns are not linear in bank size, beyond a small, diverse set, tightening a rule tends to help more than adding examples.

C

Evaluation Pipeline Details

Evaluating Text-to-SQL correctness is fundamentally harder than string matching: Syntactically different queries can produce identical results, while structurally similar queries can return different data due to a single wrong filter value. Standard benchmarks sidestep this with execution accuracy (run both queries, compare outputs), but this is insufficient for production-scale schemas where result-set equality is sensitive to row ordering, null handling, and floating-point precision. C.1

Comparison with Standard Benchmarks

Spider (Yu et al., 2018) uses exact set match (ESM): The gold and predicted SQL are decomposed into components (SELECT, WHERE, GROUP BY, ORDER BY) and compared structurally. ESM is brittle, it rejects semantically equivalent queries with different syntactic structure (e.g., a subquery reformulated as a JOIN, or an EXISTS replaced by IN). BIRD (Li et al., 2023b) improves on this by using execution accuracy (EX): Both queries run against the database and the result sets are compared. However, EX treats any difference in output as failure, even when differences arise from column ordering, alias naming, or equivalent NULL handling. Our evaluation addresses both limitations. We do not rely on structural matching alone (unlike ESM), since our queries have many valid reformulations (CTEs vs. subqueries, different join or-

derings, aliasing choices). We go beyond naive execution comparison (unlike EX) by incorporating column alignment, order-invariant comparison, AST canonicalization, and an LLM judge for semantic equivalence. C.2

Three-Tier Pipeline

A single evaluation method is insufficient for normalized financial schemas. Consider a query retrieving Apple’s annual revenue: the gold SQL joins through ciqCompany, ciqSecurity, ciqTradingItem, ciqFinPeriod, ciqFinancialData, and ciqDataItem with specific filter predicates. A correct prediction might use different aliases, reorder joins, wrap part of the logic in a CTE, or select columns in a different order, all the while producing identical results. -- Gold: two aliases of ciqCompany via inline joins SELECT target.companyName, acquirer.companyName FROM ciqTransactionV2 tr JOIN ciqCompany target ON tr.companyId = target.companyId JOIN ciqTransactionToCompanyRelV2 cr ON ... JOIN ciqCompany acquirer ON crel.companyId = acquirer.companyId WHERE target.companyId = 24937 -- Generated: equivalent result, -- CTE + different join topology WITH TargetCo AS (SELECT companyId FROM ciqCompany WHERE companyId = 24937) SELECT c.companyName, acq.companyName FROM ciqTransactionV2 tr JOIN TargetCo tc ON tr.companyId = tc.companyId JOIN ciqCompany c ON tr.companyId = c.companyId JOIN ciqCompany acq ON crel.companyId = acq.companyId ...

Both return identical rows, but the CTE node and rearranged self-join aliases produce an irreducible tree difference. No single comparison method handles all such cases, motivating a cascading pipeline where each tier catches what the previous one misses. The first tier parses both gold and generated queries into abstract syntax trees using sqlglot and applies canonicalizing transformations: alias normalization (replacing all aliases with canonical names derived from the underlying table), date function standardization, JOIN reordering (sorting inner joins by table name), and predicate normalization (canonicalizing commutative predicates and IN-clause value ordering). After canonicalization, tree edit distance determines whether the remaining differences are semantically meaningful. For selfjoins, all alias permutations are tested, tractable for our schema’s typical 2–3 self-join aliases. Queries that are structurally equivalent modulo these transformations are matched without execution. When AST matching fails, the second tier executes both queries against the production database

and compares outputs. Execution failures score 0 immediately. For successful queries, gold columns are matched to generated columns via select clauses from SQLs or case-insensitive name matching, falling back to data-type and value-distribution similarity for ambiguous cases. Values are compared with order-invariant set equality, and numeric comparisons apply a relative tolerance of 10−6 for floating-point differences. When output tables still differ, the third tier invokes an LLM judge that receives the original question, gold and generated results. The judge determines semantic equivalence in context, handling cases where column names differ but values match, results are formatted differently. We manually validated 50 randomly sampled judge decisions and found 94% agreement with human annotators (47/50 correct, 2 false positives, 1 false negative).

D

Baseline Adaptation Details

All baseline methods were originally evaluated on Spider (Yu et al., 2018) or BIRD (Li et al., 2023b), which feature flat schemas with 1–3 joins per query. Adapting them to our heavily-normalized financial databases (6+ joins on average, 16+ reference tables) required both infrastructure decisions and architectural choices. This appendix documents the adaptations made and the rationale behind each. D.1

Shared Infrastructure

All baselines receive the same infrastructure that does not constitute a methodological advantage: • Database connection: Read-only Snowflake access to execute SQL against the production schema. • Entity linking: Company name disambiguation via an external service (Kensho Link) that resolves natural-language entity references (e.g., “Nvidia” → companyId=32307). This is provided to methods that operate within a pipeline where entity resolution is a separate preprocessing step. Methods that instead explore the database directly (MACSQL) achieve equivalent resolution through interactive queries. • CTE injection: Gold SQL queries reference materialized CTEs for currency conversion and temporal normalization. These CTEs are injected into both gold and predicted queries

at evaluation time, they are infrastructure specific rather than model output. • Post-processing: All baselines use the same safety validator and temporal postprocessor as FLINT (Section 3.5). • Evaluation: Three-tier evaluation (heuristic table comparison, SQL AST matching, LLM judge) applied uniformly to all methods. D.2

Direct Prompting (Zero-Shot and Few-Shot)

To establish a floor for what the LLM can achieve without any Text-to-SQL-specific architecture, we evaluate two direct prompting configurations: • Zero-shot: The full schema (all tables and columns with descriptions) and entity IDs from the external linking service are provided as prompt context. The model generates SQL in a single pass with no examples, no lookup, no schema linking, and no reflection. • Few-shot (k=3): Same as zero-shot, plus k=3 similar examples retrieved from the query bank (question and SQL only, no intermediate reasoning traces). Both configurations use the same entity linking service as FLINT, ensuring company names are resolved to correct integer IDs. The gap between few-shot and FLINT isolates the contribution of the domain-specialized pipeline (lookup agent, schema linking, domain-specific prompts, reflection). D.3

CHESS

CHESS (Talaei et al., 2024) is a multi-agent Text-to-SQL system that achieves state-of-the-art accuracy on BIRD through schema linking, candidate generation with self-consistency, and unit-testbased selection. We use the original CHESS codebase with minimal modification: • Replace SQLite execution with Snowflake connection. • Provide schema documentation as column-level descriptions (matching the format CHESS expects from BIRD’s database_description files).

• Entity linking is not provided. CHESS handles entity resolution through its own schemavalue matching and column-value retrieval modules. • Evaluate in both single-candidate (k=1) and full self-consistency (k=10 candidates with unit-test selection) configurations. All core CHESS components are preserved; the only change is the execution backend (SQLite → Snowflake). D.4

DAIL-SQL

DAIL-SQL (Gao et al., 2024) introduces questionSQL example selection via token-level similarity and a structured prompt format that separates schema representation from in-context examples. DAIL-SQL is a prompt-engineering method without its own pipeline infrastructure. We implement it within our pipeline, providing: • Schema linking via embedding-based retrieval to identify relevant tables. • Lookup agent for reference-table constraint resolution (shared infrastructure). • Entity linking for company name disambiguation (shared infrastructure). • Similar query retrieval from a query bank (the DAIL-SQL “example selection” step). • DAIL-style prompts: generic schema linking instructions and SQL generation prompts without domain-specific rules (no currency conversion hints, no period-type defaults, no metric disambiguation guidance). • No reflection (zero retry attempts). • No structural validation. We preserve DAIL-SQL’s prompt-level contribution (generic prompts + example selection) while providing the infrastructure necessary to operate on our schema, without entity linking, no method can resolve company names against a 20 millioncompany table. This gives DAIL-SQL a stronger starting point than it would have with its original prompts alone, making our comparison conservative.

D.5

DIN-SQL

DIN-SQL (Pourreza and Rafiei, 2023) decomposes Text-to-SQL into sub-tasks: schema linking, query classification (easy/medium/hard), and difficultyaware SQL generation with self-correction. Like DAIL-SQL, DIN-SQL is a promptengineering approach. We implement it within our pipeline: • Same shared infrastructure as DAIL-SQL (entity linking, lookup agent, similar query retrieval). • DIN-SQL-style prompts: step-by-step schema linking instructions and classification-aware generation prompts. • Self-correction with 1 reflection attempt (matching the DIN-SQL paper’s selfcorrection step). • No structural validation. We preserve DIN-SQL’s key contribution: decomposition into classification-aware sub-tasks with self-correction through the prompt structure and single reflection attempt. The shared infrastructure gives DIN-SQL the same advantages as DAILSQL, making this a conservative comparison. D.6

MAC-SQL

MAC-SQL (Wang et al., 2025) proposes a Selector → Decomposer → Refiner architecture where the Decomposer has interactive database access through a multi-turn tool-calling loop. We implement MAC-SQL as a standalone pipeline, independent of FLINT’s infrastructure: • Selector: Single LLM call that receives the full schema (all tables and columns with descriptions) and returns a pruned subset relevant to the question. No embedding-based retrieval, only pure LLM schema selection. • Decomposer: Multi-turn agentic loop with two tools: execute_sql (run intermediate queries against Snowflake, returns truncated results) and submit_final_sql (terminate with final answer). Budget of 5 intermediate queries. The model explores reference tables, verifies column values, and tests subqueries before submitting.

• Refiner: If the final SQL fails execution, a single-shot LLM call attempts to fix it given the error message. Maximum 2 retry attempts. • No entity linking, no lookup agent, no similar query retrieval, no schema linking embeddings. The Decomposer’s tool-calling loop serves as a functional equivalent to entity linking. Instead of resolving entities via an external service, the model queries the database directly (e.g., SELECT companyId FROM ciqCompany WHERE companyName ILIKE ’%Nvidia%’). This means MAC-SQL does not require Kensho Link infrastructure, achieving entity resolution through interactive exploration. D.7

Chase-SQL

Chase-SQL (Pourreza et al., 2025) generates multiple candidate SQL queries using diverse reasoning strategies and selects the best one through pairwise comparison. We implement Chase-SQL as a standalone pipeline: • Three generators (run in parallel): 1. Divide & Conquer CoT: Decomposes the question into sub-problems, solves each, combines via CTEs. 2. Query Plan CoT: Reasons through execution plan steps (scan → filter → join → aggregate → project). 3. Online Synthetic: Generates 3 similar question-SQL examples on the fly, then uses them as few-shot context. • Query Fixer: Failed candidates receive up to 2 retry attempts with the error message. • Pairwise Selection: All executable candidates are compared in a tournament. Each pair is evaluated by an LLM judge that sees both queries, their results (first rows), and the original question. The candidate with the most wins is selected. • All generators receive the full schema as text. No schema linking, no entity linking, no lookup agent. We preserve all three generation strategies and the pairwise selection mechanism from the paper. Unlike MAC-SQL, Chase-SQL has no mechanism for

interactive database exploration or entity resolution, instead generating SQL in a single pass, making it more vulnerable to reference-table resolution failures. D.8

FlexSQL

FlexSQL proposes a five-stage pipeline: schema exploration, plan generation, plan clarification, SQL generation, and self-evaluation with retry. We implement all five stages with entity linking and Snowflake execution. In the 3-shot configuration, retrieved examples (with schema links and ID resolutions) are injected at plan generation to inform table selection and join paths. Self-evaluation reexecutes and retries once on failure. D.9

ReFoRCE

ReFoRCE (Deng et al., 2025) uses iterative selfrefinement: An exploration phase generates diagnostic queries against the database, then the model iteratively refines SQL using execution feedback (up to N =5 iterations, early stop on success). We implement the full exploration and refinement loop with Snowflake execution. In the 3-shot configuration, retrieved examples with ID resolutions are injected into the initial generation prompt. D.10

with generic prompts” against “FLINT with domain-specific prompts.” A literal port of these methods (standalone, no infrastructure) yields near-zero accuracy and provides no useful signal. We believe that the current comparison better isolates the contribution of prompt engineering.

Summary of Adaptations

Table 7 summarizes which infrastructure components each method receives. The shared infrastructure (entity linking, lookup agent) given to DAIL-SQL, DIN-SQL, and FlexSQL makes our comparison conservative. Without it, these methods cannot resolve company names or reference-table IDs at all, and would achieve near-zero accuracy. We chose to give them the strongest possible implementation to demonstrate that the gap with FLINT is due to the generation strategy and domain-specific prompt engineering, not missing pre-processing. MAC-SQL, Chase-SQL, and ReFoRCE receive no shared infrastructure because their architectures are selfcontained. MAC-SQL and ReFoRCE achieve entity resolution through database exploration, while Chase-SQL’s lack of any resolution mechanism is an inherent architectural limitation. We acknowledge several tensions in this evaluation: 1. DAIL-SQL and DIN-SQL within our pipeline: By providing entity linking and lookup, we are effectively comparing “FLINT

2. Chase-SQL without value retrieval: The original Chase-SQL uses LSH-based value retrieval to inject relevant column values into generator prompts. Our implementation omits this component, which may disadvantage Chase-SQL on entity-resolution-heavy queries. However, adapting LSH retrieval to our reference-table structure (where values are IDs rather than human-readable strings) would require non-trivial re-engineering that goes beyond the paper’s design. 3. MAC-SQL Selector: The original MACSQL uses embedding-based schema selection, while our implementation uses a single LLM call over the full schema text. On schemas of our size (18–24 tables), the LLM-based Selector may perform differently than embedding retrieval, though both achieve the same goal of schema pruning. D.11

Failure Analysis

Strong baseline methods achieve 80–90% on Spider and 60–75% on BIRD, where column values are human-readable and queries require 1–2 joins. On our schemas, the shared failure mode is inability to resolve opaque reference-table IDs. To quantify this directly, we define filter predicate accuracy: For each query requiring numeric ID filters (95% of queries), we check whether all gold filter values appear in the generated SQL. Table 8 reports this metric alongside the fraction of failures attributable to wrong IDs. FLINT resolves 94%/75% of required IDs (Financials/Transactions); without its lookup agent, ID accuracy drops to 10%/7%, identical to ChaseSQL and 93–94% of failures become ID errors. Among baselines, Chase-SQL resolves all required IDs in only 6–7% of queries. DAIL-SQL and DINSQL receive both entity linking and the lookup agent, yet resolve IDs in only 21–29% of queries. This reveals that providing correct ID candidates is insufficient, domain-specific prompts are required to select the correct value among ambiguous options (e.g., distinguishing Total Revenue from Net

Table 7: Infrastructure and architectural components per method. ∗ CHESS uses its own schema-value matching. † MAC-SQL

Ch as e-S Q

׆ ׆ LLM × Refiner ✓ ×

× × × × Fixer × ✓

Table 8: Filter predicate analysis. ID Acc.: fraction of

Method

Transactions

ID Acc.

Fail→ID

ID Acc.

Fail→ID

FLINT (full) w/o Lookup Agent

94% 10%

13% 93%

75% 7%

28% 94%

CHESS (k=10) MAC-SQL DAIL-SQL DIN-SQL FlexSQL ReFoRCE Chase-SQL

46% 30% 21% 22% 44% 24% 6%

64% 76% 83% 85% 76% 82% 96%

58% 39% 28% 29% 69% 28% 7%

61% 62% 79% 76% 46% 79% 94%

Revenue, or choosing the correct period type). The gap between FLINT and DAIL/DIN is not infrastructure access but prompt-level domain knowledge that guides ID disambiguation. • CHESS: Self-consistency cannot fix ID errors consistent across samples. All k candidates share the same incorrect filter predicates. Its schema-value matching provides partial resolution (46% ID accuracy), explaining its relative strength. • Chase-SQL: Three generation strategies converge to identical SQL when the bottleneck is what values to filter on, not how to structure the query. Without any value-retrieval mechanism, ID resolution is near-zero. • MAC-SQL: Interactive access allows discovering some reference values (30–39% ID ac-

✓ × ✓ ✓ ✓ × ×

× × × ✓ ✓ ✓ ×

curacy), but its 5-query budget is exhausted before resolving all required lookups in complex queries.

queries where all required numeric IDs are correctly resolved. Fail→ID: among incorrect predictions, the fraction where ≥1 required ID is wrong. On Financials, 64–96% of all failures are attributable to incorrect filter values. Financials

Re Fo RC E

M AC -S QL

✓ ✓ ✓ ✓ DIN × ×

ex SQ L

DA IL /

×∗ × ✓ × × × ✓

Fl

CH ES S

✓ ✓ ✓ ✓ ✓ × ×

Entity linking Lookup agent Schema linking Similar query retrieval Reflection / retry DB exploration Multi-candidate

DI N

FL IN T

Component

L

achieves equivalent resolution through interactive DB queries.

• ReFoRCE: Refinement fixes execution errors but cannot fix semantically wrong filter values, the query executes successfully with wrong IDs, so no error signal triggers refinement. • DAIL-SQL / DIN-SQL: These receive both entity linking and the lookup agent, yet achieve only 21–29% ID accuracy. The lookup agent provides candidate IDs, but generic prompts lack the domain-specific rules needed to disambiguate among similar options (e.g., which revenue metric, which period type). • FlexSQL: FlexSQL on Transactions is an exception where less than half of the errors are attributable to the ID resolution. On Transactions, FlexSQL gets the right tables only 65% of the time and right columns 46% which explains the bulk of non-ID failures. It has good ID resolution (69%) but picks wrong tables/columns, meaning its plan-based schema exploration still misses structural paths.

The ablation confirms this diagnosis: removing FLINT’s lookup agent reduces accuracy to 55%/33% (Financials/Transactions), comparable to the strongest baselines. The cleanest test holds the schema fixed: given FLINT’s own schema links, DAIL/DIN retain ∼0.97 table and 0.70–0.81 column overlap with gold, yet still score only 32–40%, so their remaining errors are wrong filter IDs, not wrong tables or columns, a value-grounding gap. Methods that prune aggressively instead lose the schema itself: MAC-SQL keeps only 0.39/0.20 of

gold columns (Fin/Trans), and ReFoRCE’s iterative refinement prunes hard enough that overlap collapses, a distinct failure mode that Table 8 does not separate. Once the correct schema is supplied, value grounding is what remains. Understanding FLINT Failures. Finally inspecting FLINT’s own Transactions schemalinking failures, confusion concentrates in two column groups. (1) The buyer/target relationship chain: the intermediate transaction-to-company and company-relationship tables with their parallel party-ID columns and relationship-type key (the multi-hop who-transacted-with-whom path) is dropped without pruning. (2) Currency/status scaffolding: the exchange-rate/currencyconversion tables and their flag columns, plus the transaction-status table is over-selected. This suggests Transactions-like schemas benefit from more explicit join-path and currency-usage guidance in the schema-linking prompt. Table 9 shows accuracy of FLINT stratified by structural complexity. Performance peaks on moderate queries (75%/70%), the system’s design point, where the standard join chain is well-covered by few-shot examples and lookup constraints. Simple queries (64%/66%) perform worse than moderate despite lower structural complexity. So we conduct a separate manual audit to understand this phenomenon. Crediting only failures that return the correct values while differing in a nonessential column/table raises effective accuracy from 68.8%/65.2% to ∼85%/∼76%. We call this a presentation-only failure. The residual simplequery gap is not under-specification: of the eight simple-query failures on Financials, about half are presentation-only, and the genuine ones are entitylinking mistakes on otherwise-trivial questions and a tendency to over-engineer them (adding spurious currency conversion or extra joins). This over-engineering is the flip side of the robustness that helps on complex queries. Complex queries (54%/52%) degrade from compounding errors across 7+ joins. D.12

Latency Analysis

As a deployed production system, FLINT’s end-toend latency is a critical design constraint. Figure 5 compares median query latency across all methods on a logarithmic scale. FLINT’s median of 10–20s enables interactive production deployment beating all the compared

Table 9: FLINT accuracy (%) by query difficulty. Financials Difficulty

Transactions

n/N

Acc.

n/N

Acc.

Simple (1–3 joins) Moderate (4–6 joins) Complex (7+ / CTE)

14/22 107/142 31/57

64 75 54

25/38 47/66 18/34

66 71 53

Overall

152/221

69

90/138

65

Text-to-SQL systems. CHESS’s latency is driven by sequential candidate generation and database exploration, making it infeasible for real-time use.

E

Ablation Details

Each ablation disables exactly one component while keeping the rest of the pipeline intact. This section specifies the experimental condition for each. Lookup Agent. The dynamic Lookup Agent is replaced with a static mapping of the 100 most common Financials IDs and 50 most common Transactions IDs, covering about 78% of queries.. These mappings are injected into the generation prompt as pre-resolved constraints. This measures whether question-specific constraint resolution adds value over comprehensive static knowledge. Schema Linking. The schema linking stage is bypassed entirely. The SQL generator receives the full schema (all tables and columns) without intermediate table or column selection. All other components (lookup, retrieval, entity linking, reflection) remain active. Schema linking ablation condition. When schema linking is removed, the model must select among hundreds of columns from the full schema text. Table overlap remains high (0.96) because the lookup constraints and retrieved examples still guide table selection implicitly, but column selection degrades. Few-Shot Retrieval. The similar query retrieval node returns zero examples. Schema linking and SQL generation operate in zero-shot mode—no exemplar queries are provided. The lookup agent still provides reference-table constraints, isolating the contribution of structural templates. Reflection. The first generated SQL is submitted without retry regardless of execution outcome.

Median Latency (seconds)

Financials Transactions

40m 23m

1000s 4m 92s

100s 34s

20s

21s 22s

26s

FLINT

DAIL-SQL

DIN-SQL

30s 10s

98s

27s

40s 36s

45s 41s

Chase-SQL

MAC-SQL

3m 2m

10s

ReFoRCE

FlexSQL

CHESS (k=1)

CHESS (k=10)

Figure 5: Median end-to-end query latency (log scale). FLINT achieves 10–20s median, almost 2× faster than the next-fastest baseline on Financials and almost 100× faster than CHESS.

On Financials, fewer than 1% of queries trigger reflection, so this ablation has minimal effect. On Transactions, 16% of queries initially fail execution, providing the opportunity for reflection to recover.

F

(3) Structural, not answer-level: the high structural similarity reflects a shared join backbone (a common multi-table chain plus currency handling) by design; the content-aware measure, which is the leak-relevant one, shows essentially no nearduplicates on Transactions.

Query Bank Overlap Analysis

We test whether evaluation accuracy could be explained by similarity between the retrieval query bank and the evaluation set. For each evaluation query we compute the normalized tree-edit distance (TED) to its nearest query bank neighbor over canonicalized sqlglot ASTs, in a contentaware mode where table/column names and literals contribute to the distance (the leakage-relevant measure). Overlap is reported as 1 − TED.

G

Generalization to Other Domains

To test whether FLINT’s architecture transfers beyond finance, we evaluate on two external benchmarks spanning various scientific and industrial domains. For each, we adapt FLINT with only configuration changes, i.e. schema documentation, a query bank for retrieval, and domain-specific prompt rules, without any modification of the pipeline code. Our code is available at https: //github.com/kensho-technologies/flint. Table 10: Nearest-neighbor content-aware overlap (1 − SM3-Text-to-Query (Sivasubramaniam et al., TED) from each evaluation query to the query bank. 2024) is a medical benchmark over Synthea synthetic patient data (19 PostgreSQL tables, Dataset Median ≥90% match Closest (min TED) SNOMED/LOINC/RxNorm codes as opaque idenFinancials 0.78 10.6% 0.023 tifiers). The high few-shot baseline reflects strucTransactions 0.62 0.0% 0.111 tural simplicity: 68% of questions are single-table. FLINT’s gains concentrate on the harder slices, Three points show overlap does not drive ac- i.e. multi-table-join accuracy reaches 97% and unanswerable-question detection 97%, versus 90% curacy. (1) Control: baselines given the same bank (few-shot k=3, and DAIL/DIN/FlexSQL/Re- and 42% for few-shot. FoRCE via the same retrieval) still trail by ∼20–30 ScienceBenchmark (Zhang et al., 2023) provides points; if overlap conferred accuracy they would a harder test: three PostgreSQL databases from inherit it. (2) Disjoint by construction: bank ex- distinct scientific domains where even few-shot amples use a fixed canonical entity and share no achieves only 50–82%. CORDIS (EU research question with the evaluation set, so even a struc- funding, 19 tables) requires navigating a complex turally similar example filters on different entities/- many-to-many schema linking projects, institumetrics/dates and is not answerable by copying. tions, and funding programmes. OncoMX (cancer

Execution Accuracy (%)

Zero-shot

FLINT

Few-shot (k=3)

90 70 50 30 CORDIS

OncoMX

SDSS

Figure 6: Accuracy on ScienceBenchmark (Zhang et al., 2023): CORDIS (EU funding, 19 tables), OncoMX (cancer biomarkers, 25 tables), SDSS (astronomy, 6 tables).

H.1

All generated SQL executes through a read-only connection with row-level access controls. Queries are rejected at the gateway if they reference tables or columns outside the user’s entitlement scope. A structural validator inspects every query before execution, blocking statements that contain mutation keywords, unbounded scans without WHERE clauses, or joins exceeding a configurable fan-out threshold. Query execution is subject to per-session resource limits (timeout, memory, result-set size) enforced by the warehouse, preventing runaway queries from degrading shared infrastructure. H.2

Table 11: SM3-Text-to-Query (Sivasubramaniam et al., 2024) results. Hard subset: multi-table joins and unanswerable questions. FLINT’s gain concentrates on the hard subset (+11pp over few-shot). SM3-Text-to-Query Method Zero-shot Few-shot (k=3) FLINT

All (n=2000)

Hard (n=631)

77.8 92.3 95.7

58.2 85.3 96.5

biomarkers, 25 tables) demands correct join paths through a deep foreign-key graph connecting genes, diseases, and clinical tests. SDSS (astronomical surveys, 6 tables, 4.2M rows) tests retrieval over a wide-column photometric catalog with domainspecific type lookups. FLINT improves over few-shot in every domain, by as much as 14 points.(Fig. 6 and Tab. 11). The gains come from the same components: Retrieval demonstrates correct join patterns for the domain’s schema topology, domain rules enforce conventions (case sensitivity, operator choice, tablespecific semantics), and execution-based reflection recovers from wrong join paths by detecting empty results. No architectural changes are required; only the configuration layer adapts to each domain.

H

Deployment Considerations

FLINT serves production traffic against a multiterabyte financial database. Unlike offline evaluation, deployment has no gold queries, so the system must ensure safety and detect failures without ground truth supervision.

Query Safety

Monitoring and Quality Signals

Without ground truth, we rely on proxy signals to detect degradation. Execution failures (syntax errors, permission violations, timeouts) provide an unambiguous error signal and trigger immediate retry via the reflection loop. For semantically wrong but executable queries, we monitor result set anomalies: empty results for questions that should return data, unexpectedly large result sets, and null ratios inconsistent with the queried tables. User feedback (explicit ratings and implicit signals such as re-asking a rephrased question) provides a sparse but high-quality correctness signal. Aggregate metrics (execution success rate, median latency, and user satisfaction scores) are tracked over rolling windows; sustained degradation triggers review of the retrieval index and prompt templates.

I

Expanded Related Work

Outside of multi-agent systems, earlier promptengineering approaches tackle the problem without multi-agent orchestration. DAIL-SQL (Gao et al., 2024) uses token-level similarity for fewshot selection with a structured format separating schema from examples, while DIN-SQL (Pourreza and Rafiei, 2023) decomposes the task into subproblems with self-correction. Both methods indicate that prompt structure matters, but neither includes mechanisms for finding values that appear in the question or schema text (the Lookup Agent 3.1 in FLINT). A separate line of work focuses on schemalinking as a pre-processing step to reduce the amount of tokens presented to the SQL generator. C3-SQL (Dong et al., 2023) uses ChatGPT to identify relevant schemas through zero-reasoning. RESDSQL (Li et al., 2023a) trains a separate rank-

ing model to select relevant tables and columns to then pass a pruned schema to the generator. On normalized schemas, we have the extra challenge of not only identifying target tables (where the answer resides), but also bridge tables that are used for foreign-key traversal. FLINT addresses this by using join paths, resolved lookup constraints, and similar retrieved examples to guide bridge-table identification.

Related documents

• From Benchmarks to Production: A Text-to-SQL System for Complex Financial Data#1188202
• BudgetSchemaBench: A Budget-Swept Diagnostic for Schema Context in Text-to-SQL#1154715
• QUERY PROCESSING STRATEGIES IN THE PASCAL/R RELATIONAL DATABASE MANAGEMENT SYSTEM#933692
• Health Informatics and Medical Data Analysis#1178312
• Pengaruh Pengendalian Internal Terhadap Kualitas Laporan Keuangan Daerah#933784