Text-to-SQL Models Make Choices Without Saying So

Text-to-SQL Models Make Choices Without Saying So Yichen Xu

Shipei Lin

Chenhao Ma

University of California, Berkeley Berkeley, CA, USA yichen_xu@berkeley.edu

School of Data Science The Chinese University of Hong Kong, Shenzhen Shenzhen, China 225045017@link.cuhk.edu.cn

School of Data Science The Chinese University of Hong Kong, Shenzhen Shenzhen, China machenhao@cuhk.edu.cn

arXiv:2611.04318v1 [cs.DB] 3 Nov 2026

ABSTRACT A text-to-SQL answer can run, return a plausible number and still compute something other than what the analyst meant. This can happen when the request leaves a calculation choice open, such as whether to count rows or distinct entities, and the model settles it without saying so. Standard benchmarks cannot detect this, because they score each question against a single reference interpretation. We introduce SpecBridge, which tests such a choice by varying it alone: it poses three otherwise identical requests, one leaving the choice open and one per interpretation, and scores each answer by running it. The databases are real, and the choices are ones that made models fail on existing benchmarks. On 204 hand-validated tests, four language models usually compute each interpretation when asked for it, yet for the open request return both in only 15–58% of the tests where they can, often without mentioning the choice. The gap lies in reporting, not in computing. Benchmarks even penalize the alternative: on BIRD questions where such a choice changes the answer, over two in five of the answers marked wrong are right under the other interpretation. Because the gap lies in reporting, a system can narrow it without a stronger model: it first decides whether a request leaves a choice open, then answers each interpretation in its own call. On our tests and on AMBROSIA, an existing benchmark of ambiguous questions, this approach raises the share of open requests that get every interpretation by 29– 63 points over the single answer that production assistants give, without lowering precision on clear requests. It also needs fewer user turns than asking a clarifying question.

KEYWORDS text-to-SQL, database interfaces, evaluation, controlled tests, ambiguity, language models Artifact availability: the code, data and other artifacts are available at https://github.com/Xyc-arch/specbridge-artifact.

1

INTRODUCTION

Analysts increasingly ask for numbers in natural language and get them from a text-to-SQL system, often without reading the SQL. For these users a query that fails is safe, because the error is visible. The dangerous case is a query that runs and returns a plausible number, computed differently than the analyst meant. Small choices in how a number is computed can change a published finding. An influential study reported that high public debt slows growth. It first averaged growth within each country, then across countries, so a country with one high-debt year counted as much as a country with many. With corrected data and each year counted equally, high-debt countries grew 2.2% a year, not −0.1% [11]. Table S3 lists

Request on sakila_1: “For each film rating, what is the average rental price per day?” interpreted as A

interpreted as B

Interpretation A: mean of daily prices

Interpretation B: ratio of two averages

SELECT rating, AVG(rental_rate / rental_duration) FROM film GROUP BY rating

SELECT rating, AVG(rental_rate) / AVG(rental_duration) FROM film GROUP BY rating

→ G 0.668, PG 0.642, R 0.662

→ G 0.597, PG 0.600, R 0.616

Request set (shortened) and the answers counted correct specific, A “the average of each film’s daily price” A specific, B “average price divided by average duration” B open “average rental price per day” A or B, choice named Four models’ responses to the open request Haiku Sonnet returns A ✓ ✓ names the choice ✗ ✓ also returns B ✗ ✓

GPT-Sol ✓ ✗ ✗

GPT-Luna ✓ ✓ ✗

Figure 1: One request, its two correct interpretations (shaded: the SQL that differs), its request set, and four models’ responses.

more cases across six domains. A text-to-SQL system makes such choices on many requests, and the result alone does not show them. We call such a choice, settled without telling the user, a silent choice. The answer is correct for one interpretation, so checks that look for wrong SQL [2, 5] do not catch it. Take “the average rental price per day” for each film rating (Figure 1). It can be computed in two ways, which we call interpretations. Interpretation A averages each film’s daily price, and interpretation B divides the average price by the average rental duration. For two films priced at $6 for two days and $6 for six days, A gives ($3 + $1)/2 = $2.00 per day and B gives $6/4 = $1.50. Both are simple SQL, and on Spider’s sakila_1 database the two interpretations differ for every rating. We test four models: Claude Haiku 4.5, Claude Sonnet 5.5, GPT-6 Sol and GPT-6 Luna (Haiku, Sonnet, GPT-Sol and GPT-Luna below). For the request as worded, which leaves this choice open, all four return A. Sonnet and GPTLuna say that there is a choice, but only Sonnet also gives B, the ratio of two averages. A reliable text-to-SQL system must do three things: ➊ compute the interpretation that a specific request asks for, ➋ report a choice that a request leaves open together with the other interpretation, and ➌ stay quiet about a choice that the wording already settles. Production assistants handle open choices by stopping. Snowflake’s Cortex Analyst rejects a question its classifier

Yichen Xu, Shipei Lin, and Chenhao Ma

Table 1: SpecBridge and the two closest benchmarks of ambiguous text-to-SQL. Schema: synonym or split columns and tables; wording: scope, attachment, vagueness; calculation: ties, counting units, missing values, aggregation level. Possible: AMBROSIA’s data allow it, but its scores do not (Section 5).

Where the choices lie Benchmark databases, unmodified Scores the explanation Separates computing from reporting Penalizes raising a settled choice

AmbiQT [3]

AMBROSIA [29]

SpecBridge

schema

wording

calculation

× × × ×

× × possible ✓

✓ ✓ ✓ ✓

marks ambiguous [32], Databricks’ Genie asks a clarifying question where its authors’ instructions say so [6], and Microsoft’s Fabric data agent leaves it to instructions its authors write [23]. When these assistants do answer, they return one interpretation and state it [36]. By default, none of them meets ➋. Benchmarks measure ➊–➌ only in part. Spider and BIRD measure only ➊. They score each question against one reference, even when the question leaves a choice open. A model that guesses the authors’ interpretation thus scores as well as one that reports the choice. Ambiguity benchmarks [3, 29] add ➋, and AMBROSIA also adds ➌. Three gaps remain (Table 1). First, both create their choices: AmbiQT edits Spider’s schemas, and AMBROSIA generates synthetic databases. These choices lie in the schema or the wording, such as which column a word means. Choices in how a number is computed, as in the debt study, are not among them. Yet such choices arise naturally in BIRD and Spider questions, and the references settle them by convention. Second, both score only the SQL a model returns. A user who does not read SQL learns of a choice only if the model says so. Third, their scores do not show why a model misses an interpretation. The model may be unable to compute it, or it may compute it but not report it. The first calls for a stronger model; the second does not. Our key idea is a controlled test: three requests that differ in the choice and in nothing else. Two are specific, one asking for interpretation A and one for B. The third, the open request, leaves the choice open. With the two reference queries, the three requests form a request set (Figure 1). All three use the same prompt. It asks the model to state any assumption that the request does not settle and to give SQL for other interpretations. We score the SQL by execution and the explanation by a judge model. Because only the choice differs, the test separates three kinds of failure. A wrong answer to a specific request is a failure to compute (➊). On the open request, missing an interpretation that the model computes when asked for it alone is a failure to report (➋). Raising the choice on a specific request is a false alarm (➌), because its wording settles the choice. Building request sets at scale is hard for three reasons, and our pipeline, SpecBridge, addresses each in turn. First, a specific request must state every requirement of its query, so that a wrong answer means a failure to compute. BIRD and Spider questions often leave a requirement unstated, so they cannot serve as specific requests. SpecBridge therefore generates new SQL queries on unmodified benchmark databases, and code describes each query exactly in

words (Section 2.2). Second, a test reflects real use better when its choice is one that actual questions leave open, but no benchmark lists such choices. SpecBridge finds them in benchmark failures. When a model fails a BIRD or Spider question but answers correctly from an exact description of its reference SQL instead, the wording is to blame, sometimes because it leaves a choice open (Section 2.3). Third, the three requests must differ only in the choice, but when a model writes them, it often changes something else too, such as a filter. SpecBridge therefore has the model write all three from one plan that lists what they share, and checks every set with code and two reviewer models (Section 2.4). Two authors then reviewed the 307 sets that pass, without seeing model results, and kept 232. Results use the 204 of them on choice types where BIRD and Spider references take both interpretations (Section 4.2). Execution, not a model, decides every accuracy score in this paper. Finding 1: Models make choices without saying so (Section 5). On 204 accepted request sets (Figure 6), the models mostly compute what is asked (➊) and almost never raise a settled choice (➌), but leave open choices unreported (➋). For the open request, they return both interpretations in only 15–58% of the sets they can compute, and they name the choice in only 38–70% of all sets. Saparina and Lapata [30] observed that models usually give one preferred interpretation of an ambiguous question. Request sets show that they can usually compute the others: in 72–94% of the sets where a model misses an interpretation, the same model computes it correctly when asked for it alone. Most misses are thus failures to report, not failures to compute. Finding 2: Benchmarks cannot see unreported choices (Section 6). When a BIRD or Spider question leaves a choice open, its reference SQL still takes one interpretation, by the convention of the group that wrote it. BIRD’s and Spider’s own authors return one row when several rows tie, but questions that Spider imported from two older datasets usually return all of them. On BIRD, 156 development questions, one in ten of all, change their answer under the other interpretation. On 79 such questions where both interpretations are valid, accepting both raises the four models’ accuracy from 25% to 57%. For these questions, a benchmark score measures agreement with the benchmark’s authors, not correctness, and cannot show whether a model reports the choice. Finding 3: Deciding first makes reporting more reliable (Section 7). Because most misses are failures to report, not failures to compute, a method can return many of the missed interpretations without a stronger model. Phase 1 of our method, Decide-thenanswer, first decides whether a request leaves a choice open and then answers each interpretation of an open request in its own call. Compared with the single answer that production assistants give, it returns every interpretation for 29 to 63 points more of the open requests on AMBROSIA and our request sets, without lowering precision on clear requests. In a simulated conversation, it also needs 35.0 to 40.8 fewer user turns per 100 requests than asking a clarifying question. Once the user picks an interpretation, today’s systems and methods, such as LangChain [13] and DtP [30], rewrite the request, which usually keeps the choice but can change a formula or drop an output column. Phase 2 instead keeps the request and adds the pick as one clause, which computes 8.5 to 10.5 points more interpretations correctly.

Text-to-SQL Models Make Choices Without Saying So

Construct

Mine benchmarks

1 Generated query on insider

Catalogue column meanings, units and joins of each database

BIRD and Spider questions answered as asked and from an exact description

Step 1, per transaction: 𝑚 = AVG(shortintrt) over its sentiment snapshots Step 2, per order type: baseline 𝑣 = AVG(𝑚 ) ← the edited step Step 3: ratio 𝑚/𝑣 per transaction; the ten largest

Generate SQL with the requested depth and operations

Diagnose why the original failed; the other interpretation as an SQL edit

Describe exact steps in words, by code; a model writes the request

Choice library 110 choices in 13 types

Test

query and its description

Apply edit the SQL; the answer must change

a real example of the type

Write requests for A, for B, and an open one

Evaluate execution scores the answers

Figure 2: The SpecBridge pipeline. In summary, we make three contributions: (1) request sets, controlled tests that change one calculation choice and so separate computing from reporting, with a generator that builds them on real databases and a library of choices mined from benchmark failures (Sections 2–4); (2) findings that models compute more than they report and that benchmarks cannot see this, on four models across our request sets, AMBROSIA and BIRD (Sections 5–6); and (3) a two-phase method that reports open choices and carries the user’s pick, tested against prompting, sampling, DtP and production behavior rebuilt from documentation (Section 7). We release the generator, the 307 request sets with the authors’ verdicts, all model responses, and a package that reproduces every number in this paper without new model calls.

2

SPECBRIDGE OVERVIEW

SpecBridge is named for its central step, which bridges an exact specification of each query, written by code, and the natural requests a user would write. It turns a database into request sets in two lanes that meet in one step (Figure 2). The first lane generates queries whose computation is known exactly. The second lane mines, from benchmark failures, the choices that real requests leave open. The last step applies one such choice to a generated query and writes the three requests. We follow two request sets through this process (Section 2.1) and then describe each step.

2 The choice, as an SQL edit

A: average of transaction means, AVG(𝑚 ) ; each transaction counts once

All 10 output rows differ; e.g. IT183513: ratio 2.04796 (A) vs 2.05000 (B) 3 Benchmark example of the same type

BIRD, california_schools: “Which active district has the highest average score in Reading?” Rank districts by their best school’s score, or by the average over their schools? (aggregation level) 4 Three requests (only the phrase that differs)

Specific, A: “average of the per-transaction averages (each transaction counted once)”

Specific, B: “over all matched sentiment snapshots (each snapshot counted once)”

Open: “the average short interest for its order type mix”

5 Checks passed

✓ no requirement lost or changed in any request (code and two reviewer models) ✓ each specific request accepted only for its own interpretation ✓ open request accepted for both interpretations by both reviewers

Figure 3: A request set on insider, from the generated query to its checks. Shaded: the SQL that differs.

for the average of per-film daily prices, the one for B asks for the average price divided by the average duration, and the open request asks for the “average rental price per day.” The rest of the wording is shared, and code checks that the two queries return different results on the database; a choice that does not change the answer is not tested there. The second example (Figure 3) is a three-step query on the insider database. It averages short-interest measurements for each transaction, compares each transaction with a baseline for its order type, and returns the ten largest ratios. The choice lies in the baseline. Interpretation A averages the transaction means, so each transaction counts once, while B averages the underlying snapshots, so a transaction with more snapshots weighs more. The open request asks for “the average short interest for its order type mix,” and all ten output rows change between the interpretations.

2.2 2.1

B: average over snapshots, AVG(shortintrt) ; each snapshot counts once

Generating queries

Two examples

A request set fixes a database and every requirement of a query but one choice. The film request of Figure 1 asks for each rating’s average replacement cost and daily rental price, and compares that daily price with the average across ratings. Only the daily price is open: AVG(price) 𝐴 = AVG(price/days), 𝐵= . (1) AVG(days) A averages each film’s daily price; B divides two averages and so gives more weight to longer rentals. The specific request for A asks

Generation starts from the database, not from a benchmark, so it works on databases nobody has annotated. One model review of the schema and data first builds a catalogue of column meanings, units and table relationships; after that, generation calls no model. A query starts from counts, sums or averages per entity, per group or overall. Later steps can compare a value with a broader average, take a share of a total, combine compatible quantities, aggregate again, or keep the top entities per group. We control two properties of each query, its arithmetic depth (the nesting of arithmetic expressions) and its step depth (the number of dependent query

Yichen Xu, Shipei Lin, and Chenhao Ma

stages). Rules on column types decide which quantities can be combined or summed, and data checks reject empty answers, constant or redundant quantities, non-positive denominators and too-small groups. The generator looks ahead and drops a partial query as soon as it can no longer reach the requested depths, so no attempt is spent on a dead end, and every delivered query has exactly the arithmetic and step depths it was asked for. Code then describes each query exactly in words. The text covers source rows, joins, unmatched rows, filters, groups, formulas, missing values, ordering, limits, ties and SQLite conventions, and the same SQL always gives the same text; we call it the exact description. Because the exact description states every requirement, a model that fails from it cannot compute the query, while a model that succeeds from it but fails from a natural request was misled by the wording. Section 2.3 builds on this contrast. Sonnet turns the exact description into one natural request using the column meanings. Claude Opus 5.5 (Opus below) checks the request for missing, changed or added requirements, and code checks how stored values are written. A failing request gets one revision with the reviewer’s notes and is dropped if it fails again.

2.3

Mining choices from benchmark failures

The choices come from real benchmark failures. For each BIRD and Spider question, we compare a model’s answer to the original question (with BIRD’s evidence) with its answer to the exact description of the reference SQL. When only the description is answered correctly, the wording caused the failure, not the model’s SQL, and we classify the reason as output columns, a different column or value, an unstated requirement, numerical precision, or an open choice. For an open choice, the other interpretation is written as an edit of the reference SQL, and both queries are executed. Opus and Sonnet judge each interpretation independently, and disagreements are kept. The library keeps choices of meaning, both open ones, where the wording fits both interpretations, and settled ones, where it selects one. For “What is the total number of molecules with double bonded oxygen?”, both reviewers accept only the interpretation in which the oxygen atom is part of the double bond, although the reference SQL counts every molecule that contains an oxygen atom and a double bond. Figure 4 groups the library by choice type. The largest group of open choices concerns which column a phrase refers to, and half of the settled ones concern the counting unit. The same choices appear in benchmark references, and the references do not agree on them. We say that a question leaves a choice open, or is open, when neither its wording nor BIRD’s evidence contains a word that settles it, such as “all” for ties or “distinct” for counts. Code reads which interpretation each reference takes, and Figure 4 counts, per choice type, the open BIRD and Spider questions and the share whose reference takes the rarer interpretation. Every case of the rarer interpretation was reviewed and only confirmed cases are counted, so these shares are conservative. For example, 12.3% of open tie questions return every tied row, and 8.6% of open questions that count over a join count distinct entities. Scoring one interpretation thus treats a convention as the only answer.

2.4

Building request sets

A request set needs a choice that matters on the data, so code first reports where a generated task’s columns have missing values, repeated join matches, distinct values or ties. Opus proposes one choice from the library and an exact SQL edit for it, and code checks that the edit applies, runs, changes the answer and can still be described exactly; a failed proposal gets one repair. Opus then writes the request set from a plan that lists the shared requirements and the one differing choice. It writes the request for A, edits it for B, and writes the open request. A library example of the same type may guide the wording, but never supplies the SQL. Opus and Sonnet, as two reviewers, check that the requests keep their requirements and judge each of the three requests against each interpretation. A set is kept only when both reviewers accept each specific request for its own interpretation alone and at least one accepts the open request for both interpretations. Five library types concern a filter, such as whether a cutoff is strict. Generated queries seldom filter, so for these types the generator adds one condition chosen from the data. Figure 4 gives an example of each type. The pipeline keeps 307 request sets on 12 databases, and 204 of them pass the authors’ review and a check against real references (Section 4.2).

3

MEASURES

We score what a model computes by execution and what it says by a model judge. We report four measures: accuracy for ➊, both returned and named for ➋, and false alarms for ➌. A specific request selects one interpretation, and the open request fits both. A choice is settled when the wording selects one interpretation. We execute every returned query and compare its result with the reference result, with numerical tolerance and with rows tied at a cut-off treated as interchangeable; an execution error scores zero, as in Spider and BIRD scoring. For benchmark questions we also report a score that allows extra or reordered output columns. For the open request, the model returns a main query and up to two alternatives. The model returns both interpretations when these queries include one with the result of A and one with the result of B. We report this share over all sets and over the sets in which the same model answers both specific requests correctly, where a miss cannot be a failure to compute. A missed interpretation is computable when the same model answers the specific request for it correctly. A model names the choice when its explanation says that the choice is open, whether or not it gives SQL for the other interpretation. A judge model, which does not know which model answered, decides whether a response names the choice and whether its stated interpretation matches its SQL. For a specific request, offering the other interpretation, or calling the choice open, is a false alarm; an alternative on another choice, which the set does not test, is not. The measures check one another. Returning both interpretations alone would reward a model that adds every alternative it can think of, and the false alarms charge for that; naming alone would reward a warning without an answer, and returning both interpretations charges for the missing query (Table 2). The scores are sound by construction. The design lets execution decide every score. Code describes each query in full (Section 2.2), and each specific request is written from that description,

Text-to-SQL Models Make Choices Without Saying So

One interpretation

Other interpretation

Choice type

Example open phrase

Aggregation level Rows or group averages?

“average standings points at races held there”

pool all records

average each race first

Strict cutoff Strict or inclusive?

“using 0.149 as the cutoff”

greater than 0.149

at least 0.149

Ties One tied row, or all?

“the top 10 telescopes by share of signals”

exactly ten rows every telescope tied with the tenth

Counting unit Rows or entities?

“how many manager award distinct managers winners”

Missing values Keep or skip missing?

“group members by their major”

no major is its own group

members without a major left out

“schools in Riverside”

the district field

the county or the city field

Referent column Which column is meant? Seven other types e.g. conditions on one record?

References taking the rarer interpretation

Found in failures (open + settled) 6+2

61

5.8% of 259

1+0

42

1.6% of 314 12.3% of 3,223

7+3

47

1.6% of 1,656

8+3 19 + 6

not readable

9 + 12 20 40 Choices

24

8.6% of 2,758

7 + 27

award records

0

Request sets

29 0 1

0 5 10 15 % of open questions

0 30 60 Sets

Figure 4: Choice types: an example open phrase with its two interpretations, choices found in benchmark failures (gray: open; white: settled), open BIRD and Spider questions whose reference takes the rarer interpretation, and accepted request sets. Table 2: Four recorded responses and their scores. Request, model Film, open, GPT-Sol Film, open, GPT-Luna Awards, open, Sonnet Districts, specific, Sonnet

Main SQL A: mean of daily prices A: mean of daily prices A: COUNT(*) A, as asked

Alternative SQL

Table 3: The data and where each is used.

Names Both False alarm

none

×

×

none

✓

×

B: COUNT(DISTINCT player_id) none

✓

✓

Use

Data

Sections

Request sets

12 databases, four each from Spider (baseball, Sakila, customers-and-invoices, Chinook), BIRD (financial, student club, thrombosis prediction, Formula 1) and LiveSQLBench (archeology, insider, crypto, alien) 481 BIRD development questions on five databases and 150 Spider development questions 100 BIRD and 50 Spider development questions, never used for mining every open BIRD and Spider question, training and development every open BIRD development question AMBROSIA test split, ambiguous and unambiguous questions

2.4, 5, 7

Mining choices

✓

Held-out test Conventions

so it states every step, the choice included. A specific request therefore has one correct result, and any other result is wrong. The open request drops only the choice phrase, so it differs from the specific requests in the choice alone. Its two interpretations are not ours: BIRD and Spider references take each of them on open questions, and we keep only choice types for which at least 1% of references take the less common one (Section 4.2). We therefore measure precision, the share of returned queries with the correct result, on specific requests only. On the open request we ask only whether both A and B are returned, and other answers never count against the model. If a choice admits a third interpretation, we do not score it; this can only lower the true share of open requests answered with every interpretation, so the gaps we report are conservative. On AMBROSIA, each ambiguous question comes with two or more interpretations (three for 30 of the 90 we test), and an interpretation is found when some returned query gives its result. Recall is the share of interpretations found, and AllFound is the share of questions with every interpretation found [29]. Precision on an unambiguous question is the share of returned queries with its one correct result, so extras lower it.

Answer changes Method

4

2.3 6 2.3, 6.1 6.2 5, 7

EXPERIMENTAL SETUP AND TEST CHECKS

This section gives the setup and checks that the tests change only the choice.

4.1

Experiment settings

Data. Request sets use 12 databases, four each from Spider, BIRD development and LiveSQLBench1 . The generated tasks request arithmetic depth 1–2 and step depth 3–6, and each database contributes 12 to 24 accepted request sets. Table 3 lists the data. Models. We evaluate Haiku, Sonnet, GPT-Sol and GPT-Luna (claude-haiku-4-5-20251001, claude-sonnet-5-5, gpt-6-sol, gpt-6-luna). Opus writes the request sets, reviews them with Sonnet (Section 2.4), reviews the natural requests that Sonnet writes for generated tasks (Section 2.2), and judges the explanations; Opus is not evaluated. Models see the schema and documentation and have 1 https://github.com/bird-bench/livesqlbench

Yichen Xu, Shipei Lin, and Chenhao Ma

no file or tool access. By default a model answers in one call, with a prompt that asks for SQL, for a brief statement of “any consequential assumption that the request and the supplied context do not settle,” and for SQL of any other supported interpretation, without inventing alternatives. Computing is not our main question, so to test whether a model can compute an interpretation at all, we give it its best chance and let it explore. It may first run up to six rounds of five queries on the database, seeing twenty rows of each result. Exploration changes little: GPT-Sol solves 59 of 100 held-out BIRD questions without it and 61 with it, so one call is a fair default. Statistics. Intervals resample tasks 2,000 times, keeping pairs together; proportions use Wilson intervals; paired comparisons use exact McNemar tests; differences are mean ± standard error. Baselines. We compare our method, Decide-then-answer with its option Decide-then-search (Section 7), with three research methods and two behaviors of production assistants. Plain uses the AMBROSIA authors’ prompt [29], Three samples runs it three times, and DtP (disambiguate-then-parse) [30] lists the interpretations and answers each. Single answer returns one query and states its interpretation, as Snowflake’s Cortex Analyst does (“We interpreted your question as. . . ” [36]); we use the main query of Plain. Clarifying qestion returns no SQL for a request it finds ambiguous and asks the user to choose, as Cortex Analyst does (“Your question is ambiguous, here are some alternatives” [36]) and Databricks’ Genie does where its instructions say so [6]. It asks wherever our first call finds a choice open, which favors it. It needs a second turn, so we test it in a separate experiment (Section 7.2).

4.2

Checks on the tests

The findings rest on tests that change only the choice, so we check the tests first (Figure 5). Code confirms that in each of the 307 sets that pass all checks, the two queries differ in one step and the two specific requests differ only in the choice phrase. Two authors then cross-checked every set without seeing model results and kept it only if a reasonable analyst could mean either interpretation. They keep 232 of the 307 sets (76%). Most rejected sets have an open request whose wording already favors one interpretation, or a choice that convention settles, such as “between,” which includes both ends in SQL and in almost every benchmark reference. We also keep only the choice types on which real references differ (Figure 4), which leaves 204 sets on aggregation level, cutoffs, ties, counting units, missing values and one negation. Every result below uses these 204 sets. The review does not drive the results. With no human filter, models return both interpretations in only 11–48% of the sets they can compute, so the gap holds without the review. The generator itself is reliable. Its rules find usable queries on 152 of the 190 Spider, BIRD and LiveSQLBench databases we tried; the others lack the repeated rows the rules need. On the 12 evaluation databases it delivers 1,198 of 1,200 requested queries, all at the requested depths, and each of its rules is needed (Figure S2); without the look-ahead, almost no attempt reaches its target. Natural requests also lose little against exact descriptions (Figure S3). Sonnet and GPT-Sol lose about two points, and Haiku and GPT-Luna gain. Writing from a plan keeps the three requests aligned. On 60 tasks

Choices proposed

865

The other interpretation changes the answer

781

Both interpretations described exactly

754

A reviewer model finds it open

698

One choice kept per task

332

Requests pass all checks

307

Both authors find it open

232

Other interpretation in ≥1% of references

204

Figure 5: What each construction stage keeps, from proposed choices to accepted request sets.

drawn at random across choice types, it keeps the shared requirements in 52 drafts, against 32 for direct generation (𝑝 = 0.0002), and passes all checks in 49 against 32 drafts. No model favors its own family. Claude models write and review the requests and Opus judges the explanations, so a Claude model might do better on them. We repeat each of these steps with GPT-Sol, from another family, and find no such advantage. When GPT-Sol writes the natural requests of 200 random generated tasks, neither GPT-Sol nor Sonnet gains on its own text. With exploration, GPT-Sol answers 395 of the 408 specific requests that Opus wrote, as many as Sonnet (396). When GPT-Sol rewrites the open requests of 60 random sets, no model returns both interpretations significantly more or less often (all 𝑝 ≥ 0.18). The Opus judge does not favor Claude models either. It credits naming the choice in 96–100% of responses that return both interpretations, and among responses that give one interpretation it credits GPT-Luna most often (27%).

5

MODELS MAKE CHOICES WITHOUT SAYING SO

A model that computes both interpretations when each is asked for usually returns only one of them for the open request, and often does not say that there was a choice. Asking for an explanation does not cause this gap. Against the same call asking only for SQL, correct answers to specific requests change by at most nine of 204 per model, and every paired interval includes zero. Nor does the explanation make models raise settled choices. On specific requests, no alternative returns the other interpretation, and at most one of 204 responses per model calls the choice open, such as Sonnet’s for a request to drop districts tied with the 10th (Table 2). Alternatives on other choices, which the sets do not test, appear in 4–20 of 204 responses per model. Models return one interpretation where two are correct. Returning both interpretations is much rarer than returning one correct answer (Figure 6). Even among the sets in which a model answers both specific requests correctly, it returns both interpretations for the open request in only 58% for Sonnet, 48% for GPT-Sol, 31% for Haiku and 15% for GPT-Luna. Accuracy hardly separates the models, whose answers to specific requests are right in 80–98% of sets, but reporting does. The larger model of each family returns both about two and three times as often as the smaller one, yet none closes the gap. When a model states its interpretation, the statement matches its SQL in 99–100% of responses. The unreported interpretation matters. In 98 of the 204 sets, the two interpretations

Text-to-SQL Models Make Choices Without Saying So

Haiku

Sonnet

GPT-Sol

GPT-Luna

Request sets, same single call AMBROSIA, single call Request sets, specific with exploration

Specific request: answer right

Haiku

Open request: answer is A or B

Sonnet

Names the open choice

GPT-Sol

States its interpretation correctly

GPT-Luna

Returns both A and B

0

Returns both, if both specific answers right 0

20

40 60 80 % of 204 request sets

100

Figure 6: Answering and communicating on 204 request sets, per model and measure (95% intervals).

return different rows, and in the other 106 the same rows with different values, so the answer differs in every set. Most misses are failures to report. Most interpretations a model misses on the open request are ones it can compute. The same model computes the missed interpretation correctly when asked for it alone in 72–94% of the missed sets (Figure 7). Figure 8 sorts every open request into five outcomes, namely both interpretations returned, the choice named but one interpretation given, one interpretation given silently although the model computes both, a specific request computed wrong, and a specific answer whose SQL does not run. For Haiku and GPT-Luna the most common outcome is the silent one, in which the model can compute both interpretations but gives one without naming the choice, and for Sonnet and GPT-Sol it is the second most common. A score that only checks whether both interpretations appear among the outputs, as top-𝑘 coverage [3] does, cannot tell these outcomes apart. Sampling does not surface the other interpretation either. On 55 randomly drawn sets, five SQL-only samples contain both interpretations in at most 14 for any model, while one explained response from Sonnet does so in 34. What is missing is a prompt to report the choice; SQL-only samples mostly repeat one interpretation. The gap is widest for ties and counting units. Sonnet, the best model, returns both interpretations for 68% of aggregationlevel sets but for only 43% of counting-unit sets and one of 22 tie sets, and no model returns both for more than one tie set (Figure 9). These choices hide in the SQL, in a LIMIT or a COUNT over a join, neither of which the request mentions. Cutoffs are the exception, because most cutoff requests give a value “as the cutoff” with no comparison word, which invites both the strict and the inclusive interpretation. The gap holds on every source, and no model returns both interpretations for more than 67% of the sets from any source. Naming a choice is not the same as giving the other interpretation. A model that says a choice is open often stops there. Among the sets in which a model names the choice and can compute both interpretations in the same call, it still returns only one interpretation in 10–65% of them, and GPT-Luna does so most often. Such a response warns the user but leaves them to ask for the other

20 40 60 80 100 Missed interpretations the model computes when asked (%)

Figure 7: Missed interpretations that the same model computes when asked for each of them alone, per model, on request sets with one call or after exploring the database, and on AMBROSIA (Wilson 95% intervals). Both interpretations returned Silent, both computable A specific answer does not run Haiku

59

14

Sonnet

Choice named, one interpretation A specific request computed wrong

90

117

GPT-Sol

35 23

97

GPT-Luna

26

0

45

25

59 80

98

50 % of 204 open requests

16 26

75

100

Figure 8: What each model does with the 204 open requests (one call with explanation). Computable: the same call answers both specific requests correctly. interpretation, a step the model could have taken itself, since it computes that interpretation when asked for it alone. Silent choices follow conventions on our sets, but not on BIRD. On our request sets, when a model settles a choice without saying so, it usually takes the interpretation that benchmark authors take. In 61 tie and counting sets, one interpretation is the convention of BIRD and Spider references (Section 6.1), namely a plain LIMIT rather than every row tied at the cut-off, and counting joined rows rather than distinct entities. The conventional interpretation is the intended one in 29 of these sets, so the design favors neither. Yet when a model’s main answer is one of the two interpretations, it is the conventional one in 76–91% of cases for every model (Figure 10). The conventional interpretation is also the simpler SQL, so these data cannot separate habit from simplicity. On BIRD’s own tie and counting questions the preference reverses: more answers match the other interpretation (90) than the reference (37; Section 6.2). Which interpretation a model takes silently thus varies with the request, and a user cannot tell which one an answer computes unless the model says so. Many misses on AMBROSIA are also failures to report. The pattern is not an artifact of our tests. AMBROSIA pairs each

Yichen Xu, Shipei Lin, and Chenhao Ma

Haiku

Sonnet

GPT-Sol

GPT-Luna

Original question

By choice type Strict or inclusive cutoff (42) Aggregation level (61)

Questions right (%)

Counting unit (47) Ties (24) By source of the database

BIRD (61) LiveSQLBench (72) 0

20 40 60 80 Both interpretations returned (%)

100

Figure 9: Open requests answered with both interpretations, by choice type and by source of the database, among the sets in which the same model answers both specific requests right (accepted sets in parentheses; Wilson 95% intervals). Other interpretation

38

Sonnet

8

15

48

12

GPT-Sol

39

12

GPT-Luna

42

4

0

80 60 40 20 0

Spider (71)

Haiku

Exact description

Spider held-out (n = 50)

100

Missing values (29)

Benchmark convention Neither interpretation

Extra columns allowed

BIRD held-out (n = 96)

25 50 75 % of 61 open requests (tie and counting sets)

10 15

100

Figure 10: Main answer to the open request on 61 tie and counting sets: the interpretation BIRD and Spider references take by convention, the other interpretation, or neither.

ambiguous question with one unambiguous question per interpretation, and Saparina and Lapata [30] observe that models usually give one preferred interpretation. We split each missed interpretation into one the model cannot compute and one it computes but does not report. On 120 ambiguous test questions per model, the model computes 49–66% of the interpretations it misses when asked the unambiguous version (Figure 7). Even when a model answers every unambiguous version correctly, it returns all interpretations for the ambiguous question in only 12–28% of questions. On the 40 vagueness questions per model, which ask which column or value a phrase means, models compute 48–56% of the interpretations they miss, so the gap also covers the referent-column choices that our request sets leave out. On both benchmarks, models leave open choices unreported, whether the ambiguity lies in wording or calculation.

Haiku Sonnet GPTSol

GPTLuna

Haiku Sonnet GPTSol

GPTLuna

Figure 11: Held-out questions: original question versus the exact description of the reference SQL (95% intervals).

6

BENCHMARKS CANNOT SEE THE GAP

BIRD and Spider score a model against one interpretation even where a question leaves the choice open, and which interpretation they take depends on who wrote the question. On held-out questions, the four models answer 49–74% of BIRD questions correctly as worded but 96–100% from the exact description of the reference SQL (Figure 11). Allowing extra or reordered output columns closes only part of this gap, and on Spider the remaining gap is smaller. The exact description gives the reference’s steps, columns and choices at once, so this gap bounds what the wording costs rather than measuring ambiguity alone. To see what it consists of, Opus assigns one cause to each development question answered correctly from the description and wrongly as worded (Figure 12). Output columns, a phrase mapped to another column or value, unstated requirements and run-to-run variation cause most failures. A second labeler from another family, GPT-Sol, given the same evidence, agrees on the cause for 53% of the questions (Cohen’s 𝜅 = 0.44); it often calls Opus’s run-to-run variation output shape and finds an open choice in 35 questions where Opus finds 16, so the figure, if anything, undercounts open choices. Output columns are a matter of form, as the answer holds the right values in a different shape, such as a name instead of an ID. Open choices, which our library collects, are one of these causes. Failures undercount them, because an answer following the benchmark’s convention is scored right; Section 6.2 measures them on every question that leaves a choice open. Other choices are about which column a word means. For “Name schools in Riverside. . . ”, filtering the district field returns six rows, the county field sixty and the city field seven. Code cannot tell from a reference which column the question’s word could also mean, so request sets leave this type to AMBROSIA’s vagueness questions.

6.1

References follow unstated conventions

When a question leaves a choice open, its reference still takes one interpretation, and the interpretation depends on the author group. For other common choices, such as join type, missing values and whether “between” includes its bounds, every author group takes the same interpretation in at least 91% of open questions. On

Text-to-SQL Models Make Choices Without Saying So

BIRD (n = 162)

Spider (n = 51)

% of development failures

Unstated requirement

4

Phrase mapped to another column

19

8

6

Open choice

Table 4: Six of the 17 BIRD questions on which all four models return the other interpretation, confirmed by two authors; BIRD scores each of these answers wrong.

17

14 8

Model added steps

10

Run-to-run variation

25

30

17

Output columns differ 1 2

Numeric precision Other

3

31

6

Figure 12: Why the original question failed, on development questions answered right from the exact description and wrong as worded, one cause per question.

Question

BIRD reference

Other interpretation

“Who is the tallest superhero?” “. . . cards in the set Coldsnap with the highest converted mana cost.” “What are the expenses of the budget with the lowest remaining?” “How many of the banned cards are white border?” “. . . normal glutamic pylvic transaminase level, how many of them are male?” “. . . diagnosed with SJS, how many of them have a normal level of total protein?”

1 hero 1 card

2 tied heroes 12 tied cards

1 expense

2 tied expenses

258 rows

89 cards

1,924 lab records

45 patients

780 lab records

37 patients

A single value moves > 50% Other change

BIRD Spider, own authors Spider, imported from older datasets Ties at an extreme: % returning one row

Count over a join: % counting joined rows

Moves 10–50%

40%

Bound inclusive 29%

Rows vs entities BIRD train

93%

92%

BIRD dev

95%

91%

Unmatched rows

Spider train

96%

95%

Missing values

Spider dev

98%

99%

0% Academic, IMDB, Yelp

9% Geo, Restaurants

Spider imported 0

50

100 0

50

100

Figure 13: Share of open questions whose reference takes the usual interpretation, by author group, for the two choices on which groups disagree, ties and counting.

ties and counting, the groups disagree (Figure 13). Asked for an extreme value, BIRD and Spider references return one row in over 90% of open questions, but the Geo and Restaurants questions that Spider imported from older datasets return one row in 21 of 247. Asked for a count over a join, BIRD references count joined rows in most open questions, and the imported Academic, IMDB and Yelp questions in none of 39. Each group keeps its convention in training and development data alike, so examples retrieved from a benchmark’s training data can carry its authors’ convention into a model’s answers. For the same wording, a model that follows BIRD’s convention is therefore scored right on BIRD and wrong on the imported Spider questions.

6.2

26%

Ties at extreme

The unstated interpretation changes answers

The choice a reference makes for an open question is not cosmetic. For each open BIRD development question, we apply the library’s edit for the other interpretation to the reference SQL wherever it gives a valid interpretation, and run both. Of 664 such questions, 156 (23%) change their answer (Figure 14). These are one in ten

13% 10% 0

20 40 % of open questions whose answer changes

60

Figure 14: Open BIRD development questions whose answer changes under the other interpretation, by choice type and by how far a single answer value moves.

of all BIRD development questions. Counting moves the answer most, often by more than half of its value. For “How many accounts in North Bohemia has made a transaction with the partner’s bank being AB?”, the reference counts joined rows and returns 2,260, while counting each account once returns 55. Ties change about a quarter of answers, often the list of returned items, and join type and missing-value filters change about one in ten. The data behind these choices are common. Every one of the 12 evaluation databases has at least two kinds of data on which an unstated choice changes an answer, such as joins that repeat rows, columns with missing values and maxima shared by several rows. BIRD scores the other interpretation as wrong. On the 79 questions checked below, accepting both interpretations raises accuracy from 25% to 57%; the difference is answers that BIRD counts as wrong SQL, although the question allows them. We ask the four models 137 BIRD development questions on which a tie, counting or join edit of the reference changes the answer, and score each answer against both queries. Two authors checked that the edit gives a valid interpretation of the question, and 79 questions pass. Of the 236 answers BIRD marks wrong on them, 99 return the other interpretation. On tie and counting questions, more answers match the edited reference (90) than the reference itself (37), and on 17 questions every model returns the other interpretation (Table 4).

Yichen Xu, Shipei Lin, and Chenhao Ma

Request: “For each film rating, what is the average rental price per day?”

Decide-then-search Three samples DtP Sonnet

1 Phase 1: one call lists the interpretations, if any are open

B: average price / average duration

2 Each interpretation is answered in its own call AVG(rental_rate / rental_duration)

AVG(rental_rate) / AVG(rental_duration)

3 Both run on the database; identical results are shown once

G 0.668, PG 0.642, R 0.662

≠ G 0.597, PG 0.600, R 0.616

4 The user sees one answer and the choice

Answer A. The request leaves open how the daily price is averaged; interpretation B gives G 0.597, PG 0.600, R 0.616. Phase 2, after the user picks B: the request, unchanged, + “Divide the average price by the average duration.”

Figure 15: Both phases on the film request. Blue and orange: interpretations A and B, as in Figure 1.

7

A TWO-PHASE METHOD FOR OPEN CHOICES

A conversation about an open choice has two moments, before the user says which interpretation they mean and after. Production assistants ask or return one interpretation before (Section 1) and rewrite the request after (Section 7.3). Our method has one phase for each moment, and each step follows from a finding of this paper. Models state their interpretation accurately and almost never call a settled choice open (Section 5), so one call can decide whether a choice is open; they compute most interpretations they miss once each is asked for alone, so each gets its own call; and a rewrite keeps the choice but loses other requirements (Section 7.3), so each interpretation is written as the full request and the pick is added as one clause, leaving the rest word for word.

7.1

GPT-Sol 6.7

60 50

6.0

40

3.0

30

8.0

20

8.7

2.6

3.1 3.0

1.0

10

1.0

1.0

1.0

0 30

40 50 60 70 Precision, unambiguous (%)

30

40 50 60 70 Precision, unambiguous (%)

Figure 16: Methods on 150 new AMBROSIA test questions per model; calls per question beside each point (95% intervals). Table 5: Decide-then-answer, Decide-then-search and the baselines of Section 4.1, measured the same way on both datasets: coverage (all interpretations returned, of 90 ambiguous AMBROSIA questions and 111 open requests), precision on clear or specific requests (%), and calls per request. Bold: best per model. AMBROSIA

Request sets

Model

Method

Cover Prec. Calls Cover Prec. Calls

Sonnet

Plain Three samples DtP Single answer Decide-then-answer Decide-then-search

26 31 30 9 32 37

49.1 49.0 47.1 65.0 62.6 54.0

1.0 3.0 8.0 1.0 3.1 8.7

64 72 10 0 78 79

94.6 92.0 18.6 98.2 96.1 89.2

1.0 3.0 8.0 1.0 2.6 6.7

GPT-Sol Plain Three samples DtP Single answer Decide-then-answer Decide-then-search

18 24 54 8 37 45

61.4 58.8 43.7 65.0 66.7 65.4

1.0 3.0 6.7 1.0 2.6 6.0

47 60 35 0 62 68

83.3 82.6 67.3 82.9 93.7 91.5

1.0 3.0 5.3 1.0 2.5 5.3

Phase 1: decide, then answer

Phase 1, Decide-then-answer, runs before the user replies (Figure 15). First, one call lists the interpretations of the request, each written as the full request with exactly one interpretation; if it lists only one, one more call answers it, and nothing else runs, so a clear request costs two calls and one query. Second, where a choice is open, each interpretation it lists, at most four, is answered in its own call. Third, every answer runs on the database, SQL that fails is dropped, and identical results are shown once. The user sees the first answer with the others as alternatives, and phase 2 starts when the user picks one (Section 7.3). An option, Decide-then-search, first searches wide in the second step, with two more calls that list the interpretations and then any that are missing, using DtP’s prompts [30], for users who must not miss an interpretation.

7.2

Plain

70 AllFound, ambiguous (%)

A: average each film’s daily price

Decide-then-answer Single answer

Phase 1 against products and baselines

We developed the method on 60 AMBROSIA questions. The final test uses 150 new test questions per model (90 ambiguous, 60 unambiguous) and 111 random request sets, with Sonnet and GPT-Sol.

More interpretations than today’s answer, at no loss of precision. Pooled over both models and paired by item, Decidethen-answer returns every interpretation for 28.9 ± 3.5 points more ambiguous AMBROSIA questions than Single answer and for 63.1 ± 3.2 points more open requests, with no loss of precision (−0.3 ± 3.7 points on AMBROSIA, +4.4 ± 2.0 on request sets), and its main answer is right on 6.3 points more specific requests. Precision holds because the first step finds no open choice in 86–90% of specific requests, which then get one answer, as from Single answer. Against Plain, Three samples and DtP, it is the most precise, by 9.4 to 19.2 points on AMBROSIA and 5.9 to 52.0 points on request sets, all significant; it covers more than Plain and Three samples, and more than DtP on request sets, by 42.8 points, where DtP’s restatements lose other requirements. Only on AMBROSIA does DtP cover more, by 8.3 points, at 19.2 points less precision and nearly three times the calls, having no step for clear questions. The first step brings the gain. Decide-then-search is DtP with the first step added, so their difference isolates that step: the same AllFound (−1.1 points) and 14.3 points more precision. The

Text-to-SQL Models Make Choices Without Saying So

gain is not from returning fewer alternatives. With DtP cut, per question, to the number of interpretations that the first step lists, this budget finds every interpretation on 7.6 ± 2.0 points more ambiguous questions when spent where the first step finds a choice than at random, and Decide-then-answer still finds every interpretation on 15.0±2.9 points more ambiguous questions at no lower precision. Decide-then-search finds every interpretation for 7.2 points more ambiguous questions but is 5.0 points less precise, at 2.3 times the calls, so it suits users who must miss no interpretation. Two-turn experiment: fewer user turns than asking. Clarifying qestion needs a second turn, so we test it separately. A simulated user has one interpretation in mind and, when the first turn does not show it, states it in one more turn; the model then answers the specific request, the same second answer for every method. Per 100 requests, with one in five AMBROSIA questions ambiguous, Decide-then-answer needs 35.0±3.3 fewer user turns than Clarifying qestion and 14.9 ± 2.8 fewer than Single answer, and 40.8 ± 2.4 and 22.1 ± 2.0 fewer on request sets. It also ends right on 9.1 ± 2.2 and 12.6 ± 2.5 more AMBROSIA requests and on 3.4 ± 0.9 and 6.1 ± 1.1 more request-set requests, because an answer shown in the first turn needs no second answer that can go wrong, at 1.1 to 1.5 more calls per request.

7.3

Phase 2: keep the request, add the pick

When the user picks an interpretation, today’s systems rewrite the request. Cortex Analyst rewrites each follow-up into a selfcontained question [40], the Fabric data agent rephrases each question [23], LangChain’s ConversationalRetrievalChain [13] and LlamaIndex’s CondenseQuestionChatEngine [21] condense each follow-up into a standalone question, and DtP writes each interpretation as a short question. Each rewrite is a new copy that can lose a requirement, so phase 2 keeps the request and adds the pick as one clause, leaving the rest unchanged. Request sets test this, as their three requests differ only in the choice. For each interpretation of a request set, the same SQL prompt receives one of four texts: (a) the original specific request; (b) the open request rewritten by the model as one interpretation with DtP’s prompt, told the intended interpretation; (c) the open request unchanged, followed by a one-sentence description of the interpretation; (d) the open request and the interpretation condensed by the model into a standalone question with LangChain’s prompt [13]; we report 363 interpretations drawn at random from the 204 sets, with all four models and one SQL prompt. Keeping the request and adding one clause computes 80.7% of the interpretations correctly, as many as the original specific requests (𝑝 = 0.57, exact McNemar; Figure 17). Both rewrites compute fewer, 8.5 points fewer for the interpretation rewrite (𝑝 = 0.0004) and 10.5 for the standalone question (𝑝 < 0.0001), and the loss is largest for Haiku and GPT-Luna. The rewrite seldom loses the choice itself. Of the interpretations answered correctly from the original request and wrongly from the rewrite, the judge finds the choice lost in only 12%; the rest lose another requirement, most often a formula, an output column or a join (Figure 17, right). The same holds on real questions. We repeat the comparison on 85 BIRD development questions where one library edit of the reference SQL changes the answer. A simulated user follows each

Original specific request Open request + one clause

Rewritten as one interpretation Rewritten as a standalone question Formula

Haiku

Output columns Sonnet GPT-Sol GPT-Luna Pooled 40 60 80 100 Interpretations computed right (%)

Join The choice Filter Grouping Other 0 20 40 Rewrite failures (%)

Figure 17: Editing versus rewriting a request (363 interpretations). Left: interpretations computed right from four request texts. Right: what the rewrite lost, by kind.

question with one sentence that settles the choice, such as “If several rows tie at the cut-off, return all of them,” and the edited reference is the gold answer. Over 170 question–model pairs with Sonnet and GPT-Sol, adding the clause beats the standalone question, 58.2% against 47.6%. As on our request sets, the rewrite keeps the clarified choice and loses something else; the gap is largest for ties.

8

IMPLICATIONS

Systems could answer open choices instead of asking. Deciding first and answering each interpretation in its own call adds the other interpretations at no loss of precision and saves the user turns that asking costs (Section 7). Reporting could be a training target. The models we test already compute both interpretations of most requests; what they mostly lack is the habit of reporting the second one. Sonnet, the strongest model, still misses it in four of ten sets where it can compute it, so training may help more than scale, and request sets could supply the signal. Each interpretation has its own executable reference, so a reward can credit an open answer only when it returns both interpretations and charge a specific answer for every alternative it adds, which rewards reporting but not guessing. Benchmarks could reward stating assumptions. Current benchmarks score a single answer per question, which suits questions with one correct interpretation. For a question that leaves a choice open, a complementary practice is to let the model state the assumption it made and the alternative it set aside, and to credit both when each matches its own reference. Request sets make this possible because each interpretation has its own reference, so a benchmark can report accuracy on specific requests, reporting on open requests, and restraint on settled ones.

9

RELATED WORK

Natural language interfaces to databases. NaLIR [18] fixed “easy” ambiguities quietly and, for each “hard” one, explained its interpretation and offered several for the user to choose from. Its ambiguities lie mainly in mapping words to schema elements and in how phrases attach. Our results show that the alternatives NaLIR

Yichen Xu, Shipei Lin, and Chenhao Ma

asked users to choose among are now cheap to compute; what is missing is the report. Our method applies NaLIR’s interaction to calculation choices and lets execution decide which are “hard”. Evaluation. Spider tests generalization across schemas [42]; BIRD adds large databases and external knowledge [19]; KaggleDBQA uses realistic databases and documentation [14]; Spider 2.0 tests enterprise workflows [16]. SParC and CoSQL add context and dialogue [41, 43], and BIRD-INTERACT tests clarification and recovery [12]. It scores whether a system completes a task after asking the user, whereas request sets score whether it reports a choice it made without asking. TrustSQL rewards a model for abstaining when it cannot answer reliably and penalizes wrong answers [15]; request sets reward a different behavior, answering while reporting the open choice. Error detectors flag executable SQL that is wrong against one gold query [5], and verifiers turn such silent failures into abstentions [2]; a silent choice passes both, because its SQL is correct for one interpretation. Test suites [44] catch wrong queries whose results match by chance; our concern is the opposite error, in which a correct query for another valid interpretation is rejected because the reference fixes one interpretation. Pourreza and Rafiei re-evaluated top models on Spider and BIRD by hand and found a perfect score out of reach “due to the multiple interpretations that can be derived from the provided samples” [28], and Wretblad et al. found noise in BIRD’s questions and references whose correction changed which method ranks first [39]. We measure the first problem by construction rather than inspection and keep possible reference errors apart from open choices, because a wrong reference and a valid but different one need different fixes. Ambiguous requests. The closest benchmarks are AmbiQT [3] and AMBROSIA [29] (Table 1). AmbiQT makes Spider questions ambiguous by changing the schema, with synonym columns or tables, split tables and precomputed aggregates, and checks whether both gold queries appear in the top five outputs. AMBROSIA has human-written questions with scope, attachment and vagueness ambiguity on generated databases. Both locate ambiguity in the schema or the wording. The choices behind the conventions of Section 6.1, such as ties, counting units, missing values and aggregation level, lie in the calculation, outside both taxonomies. Outside SQL, AmbigQA asks for every plausible answer to an ambiguous question [24], and conversational search studies when a system should ask a clarifying question [1]. Generation, SQL methods and semantic layers. DBPal [38] and GAZP [45] generate training pairs for a target database; we also start from the database, but for evaluation. RAT-SQL, PICARD, DINSQL, DAIL-SQL, CHASE-SQL and Alpha-SQL [9, 17, 26, 27, 31, 37] improve how the SQL is generated. LookML has developers write each measure “once, in one place” [10], the dbt Semantic Layer centralizes metric definitions [7], and Cortex Analyst generates SQL from them [35]. A layer settles the choices its authors foresaw, usually a measure’s formula and counting unit; per-request choices, such as ties or the rows a ratio includes, stay open, and these are the choices our method finds and answers for the user.

10

CONCLUSION

Text-to-SQL models make choices without saying so. Request sets, which change only one calculation choice, make this visible. Four

models compute both interpretations of most sets when each is asked for, yet return both for the open request in only 15–58% of those sets. Benchmarks cannot see the gap, because their references take one interpretation by conventions that differ by author group; over two in five answers BIRD marks wrong on such questions give the other interpretation. A method that decides first whether a choice is open and answers each interpretation in its own call adds what Single answer drops at no loss of precision, needs fewer user turns than asking, and keeps the request when the user picks. The models already hold the other interpretation; what is missing is a step that shows it to the user, and a benchmark that asks for it and rewards it. Until then, a plausible number from a text-to-SQL system can answer a different question than the analyst asked, and nothing in the answer says so. We release the request sets, the library of choices, every model response and the scoring code. Limitations and future work. The generator covers a subset of analytical SQL, and the library comes from two benchmarks on SQLite, so other benchmarks may hold choice types we have not seen. The request sets cover every choice type on which real references differ, except quantifier, which the generator does not yet build, and referent column, which code cannot read off a reference; for the latter, AMBROSIA’s vagueness questions show the same gap. Each request set tests one choice at a time, so requests that leave two choices open at once are not covered. Each open request is scored on the two interpretations that real references take; further valid ones are unscored, so the gaps are conservative (Section 3). The conventions are read by rules that see one SQL form each, so their counts are lower bounds. One model family wrote the request sets, but the results hold with another writer and without the human review (Section 4.2). The BIRD rescoring uses one call per question without exploration, so its shares may shift with stronger prompting. Single answer and Clarifying qestion are rebuilt from the products’ documentation, not run on the products, and the two-turn experiment uses a simulated user who always recognizes the interpretation it means. Other SQL dialects, longer sessions and a study of how analysts act on a reported choice are our next steps.

Text-to-SQL Models Make Choices Without Saying So

REFERENCES [1] Mohammad Aliannejadi, Hamed Zamani, Fabio Crestani, and W. Bruce Croft. 2019. Asking Clarifying Questions in Open-Domain Information-Seeking Conversations. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, Paris, France, 475–484. [2] Tarfah Alrashed, Madhup Sukoon, David R. Karger, and Natasha Noy. 2026. Developing and Benchmarking Verification Algorithms to Improve Text-to-SQL Generation. Proceedings of the VLDB Endowment 19, 11 (2026), 3732–3744. https: //doi.org/10.14778/3836663.3836721 [3] Adithya Bhaskar, Tushar Tomar, Ashutosh Sathe, and Sunita Sarawagi. 2023. Benchmarking and Improving Text-to-SQL Generation under Ambiguity. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Singapore, 7053– 7074. https://doi.org/10.18653/v1/2023.emnlp-main.436 [4] Rotem Botvinik-Nezer, Felix Holzmeister, Colin F. Camerer, et al. 2020. Variability in the Analysis of a Single Neuroimaging Dataset by Many Teams. Nature 582 (2020), 84–88. [5] Shijie Chen, Ziru Chen, Huan Sun, and Yu Su. 2023. Error Detection for Text-toSQL Semantic Parsing. In Findings of the Association for Computational Linguistics: EMNLP 2023. Association for Computational Linguistics, Singapore, 11730–11743. https://doi.org/10.18653/v1/2023.findings-emnlp.785 [6] Databricks. 2025. Curate an Effective Genie Agent. Databricks documentation. https://docs.databricks.com/aws/en/genie/best-practices [7] dbt Labs. 2025. dbt Semantic Layer. dbt documentation. https://docs.getdbt. com/docs/use-dbt-semantic-layer/dbt-sl [8] Ann Furedi. 1999. The Public Health Implications of the 1995 ‘Pill Scare’. Human Reproduction Update 5, 6 (1999), 621–626. [9] Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. 2024. Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation. Proceedings of the VLDB Endowment 17, 5 (2024), 1132– 1145. https://doi.org/10.14778/3641204.3641221 [10] Google Cloud. 2025. Introduction to LookML. Looker documentation. https: //docs.cloud.google.com/looker/docs/what-is-lookml [11] Thomas Herndon, Michael Ash, and Robert Pollin. 2014. Does High Public Debt Consistently Stifle Economic Growth? A Critique of Reinhart and Rogoff. Cambridge Journal of Economics 38, 2 (2014), 257–279. [12] Nan Huo, Xiaohan Xu, Jinyang Li, Per Jacobsson, Shipei Lin, Bowen Qin, Binyuan Hui, Xiaolong Li, Ge Qu, Shuzheng Si, Linheng Han, Edward Alexander, Xintong Zhu, Rui Qin, Ruihan Yu, Yiyao Jin, Feige Zhou, Weihao Zhong, Yun Chen, Hongyu Liu, Chenhao Ma, Fatma Ozcan, Yannis Papakonstantinou, and Reynold Cheng. 2026. BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation via Lens of Dynamic Interactions. In International Conference on Learning Representations, C. Vondrick, B. Hariharan, C. Raffel, L. Pinto, D. Yang, and A. Faust (Eds.), Vol. 2026. ICLR, Rio de Janeiro, Brazil, 44285–44333. https://proceedings.iclr.cc/paper_files/paper/2026/hash/ 496b549556509bbb9770bf9d335c5800-Abstract-Conference.html [13] LangChain. 2024. ConversationalRetrievalChain and its condensequestion prompt (CONDENSE_QUESTION_PROMPT). LangChain documentation and source, langchain/chains/conversational_retrieval. https://github.com/langchain-ai/langchain/blob/v0.3/libs/langchain/langchain/ chains/conversational_retrieval/prompts.py [14] Chia-Hsuan Lee, Oleksandr Polozov, and Matthew Richardson. 2021. KaggleDBQA: Realistic Evaluation of Text-to-SQL Parsers. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, Online, 2261–2273. https://doi.org/10.18653/v1/2021.acl-long.176 [15] Gyubok Lee, Woosog Chay, Seonhee Cho, and Edward Choi. 2024. TrustSQL: Benchmarking Text-to-SQL Reliability with Penalty-Based Scoring. arXiv:2403.15879 https://arxiv.org/abs/2403.15879 [16] Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin Su, Zhaoqing Suo, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, Victor Zhong, Caiming Xiong, Ruoxi Sun, Qian Liu, Sida Wang, and Tao Yu. 2025. Spider 2.0: Evaluating Language Models on Real-World Enterprise Textto-SQL Workflows. In International Conference on Learning Representations, Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (Eds.), Vol. 2025. ICLR, Singapore, 28691–28735. https://proceedings.iclr.cc/paper_files/paper/2025/hash/ 46c10f6c8ea5aa6f267bcdabcb123f97-Abstract-Conference.html [17] Boyan Li, Jiayi Zhang, Ju Fan, Yanwei Xu, Chong Chen, Nan Tang, and Yuyu Luo. 2025. Alpha-SQL: Zero-Shot Text-to-SQL using Monte Carlo Tree Search. In Proceedings of the 42nd International Conference on Machine Learning (Proceedings of Machine Learning Research), Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu (Eds.), Vol. 267. PMLR, Vancouver, Canada, 36810–36830. https://proceedings. mlr.press/v267/li25dt.html [18] Fei Li and H. V. Jagadish. 2014. Constructing an Interactive Natural Language Interface for Relational Databases. Proceedings of the VLDB Endowment 8, 1

(2014), 73–84. https://www.vldb.org/pvldb/vol8/p73-li.pdf [19] Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2023. Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36. Curran Associates, Inc., Red Hook, NY, USA, 42330–42357. https: //doi.org/10.52202/075280-1835 [20] Jacques-Louis Lions et al. 1996. ARIANE 5 Flight 501 Failure: Report by the Inquiry Board. Technical Report. European Space Agency and Centre National d’Études Spatiales, Paris. [21] LlamaIndex. 2024. Chat Engine – Condense Question Mode (CondenseQuestionChatEngine). LlamaIndex documentation. https://developers.llamaindex.ai/ python/examples/chat_engine/chat_engine_condense_question/ [22] Mars Climate Orbiter Mishap Investigation Board. 1999. Mars Climate Orbiter Mishap Investigation Board Phase I Report. Technical Report. National Aeronautics and Space Administration. [23] Microsoft. 2026. Fabric Data Agent Creation. Microsoft Learn. https://learn. microsoft.com/en-us/fabric/data-science/concept-data-agent [24] Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. AmbigQA: Answering Ambiguous Open-domain Questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Online, 5783–5797. https://doi.org/ 10.18653/v1/2020.emnlp-main.466 [25] Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. 2019. Dissecting Racial Bias in an Algorithm Used to Manage the Health of Populations. Science 366, 6464 (2019), 447–453. [26] Mohammadreza Pourreza, Hailong Li, Ruoxi Sun, Yeounoh Chung, Shayan Talaei, Gaurav Tarlok Kakkar, Yu Gan, Amin Saberi, Fatma Ozcan, and Sercan Arik. 2025. CHASE-SQL: Multi-Path Reasoning and Preference Optimized Candidate Selection in Text-to-SQL. In International Conference on Learning Representations, Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (Eds.), Vol. 2025. ICLR, Singapore, 60385–60415. https://proceedings.iclr.cc/paper_files/paper/2025/hash/ 974ff7b5bf08dbf9400b5d599a39c77f-Abstract-Conference.html [27] Mohammadreza Pourreza and Davood Rafiei. 2023. DIN-SQL: Decomposed InContext Learning of Text-to-SQL with Self-Correction. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36. Curran Associates, Inc., Red Hook, NY, USA, 36339–36348. https://doi.org/10.52202/075280-1577 [28] Mohammadreza Pourreza and Davood Rafiei. 2023. Evaluating Cross-Domain Text-to-SQL Models and Benchmarks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 1601–1611. https://doi.org/10.18653/v1/2023.emnlp-main.99 [29] Irina Saparina and Mirella Lapata. 2024. AMBROSIA: A Benchmark for Parsing Ambiguous Questions into Database Queries. In Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37. Curran Associates, Inc., Red Hook, NY, USA, 90600–90628. https://doi.org/10.52202/079017-2876 [30] Irina Saparina and Mirella Lapata. 2025. Disambiguate First, Parse Later: Generating Interpretations for Ambiguity Resolution in Semantic Parsing. In Findings of the Association for Computational Linguistics: ACL 2025, Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Computational Linguistics, Vienna, Austria, 16825–16839. https://doi.org/10.18653/v1/2025.findings-acl.863 [31] Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. 2021. PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 9895–9901. https://doi.org/10.18653/v1/ 2021.emnlp-main.779 [32] Nipun Sehrawat. 2024. Snowflake Cortex Analyst: Behind the Scenes. Snowflake Engineering Blog. https://www.snowflake.com/en/blog/engineering/snowflakecortex-analyst-behind-the-scenes/ [33] David M. Shahian, Robert E. Wolf, Lisa I. Iezzoni, Leslie Kirle, and Sharon-Lise T. Normand. 2010. Variability in the Measurement of Hospital-Wide Mortality Rates. New England Journal of Medicine 363, 26 (2010), 2530–2539. [34] Raphael Silberzahn, Eric L. Uhlmann, Daniel P. Martin, et al. 2018. Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results. Advances in Methods and Practices in Psychological Science 1, 3 (2018), 337–356. [35] Snowflake. 2025. Cortex Analyst. Snowflake documentation. https://docs. snowflake.com/en/user-guide/snowflake-cortex/cortex-analyst [36] Snowflake. 2025. Cortex Analyst REST API. Snowflake documentation. https: //docs.snowflake.com/en/user-guide/snowflake-cortex/cortex-analyst/rest-api

Yichen Xu, Shipei Lin, and Chenhao Ma

[37] Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2020. RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 7567–7578. https://doi.org/10.18653/v1/2020.acl-main.677 [38] Nathaniel Weir, Prasetya Utama, Alex Galakatos, Andrew Crotty, Amir Ilkhechi, Shekar Ramaswamy, Rohin Bhushan, Nadja Geisler, Benjamin Hättasch, Steffen Eger, Ugur Cetintemel, and Carsten Binnig. 2020. DBPal: A Fully Pluggable NL2SQL Training Pipeline. In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data. Association for Computing Machinery, New York, NY, USA, 2347–2361. https://doi.org/10.1145/3318464.3380589 [39] Niklas Wretblad, Fredrik Riseby, Rahul Biswas, Amin Ahmadi, and Oskar Holmström. 2024. Understanding the Effects of Noise in Text-to-SQL: An Examination of the BIRD-Bench Benchmark. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 356–369. https://doi.org/10.18653/v1/2024.aclshort.34 [40] Aiwen Xu. 2024. Snowflake Cortex Analyst: Support for Multi-turn Conversation. Snowflake Engineering Blog. https://www.snowflake.com/en/blog/engineering/ cortex-analyst-multi-turn-conversations-support/ [41] Tao Yu, Rui Zhang, Heyang Er, Suyi Li, Eric Xue, Bo Pang, Xi Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Li, Youxuan Jiang, Michihiro Yasunaga, Sungrok Shim, Tao Chen, Alexander Fabbri, Zifan Li, Luyao Chen, Yuwen Zhang, Shreya Dixit, Vincent Zhang, Caiming Xiong, Richard Socher, Walter Lasecki, and Dragomir Radev. 2019. CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics, Hong Kong, China, 1962–1979. https://doi.org/10.18653/v1/D19-1204 [42] Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Brussels, Belgium, 3911–3921. https: //doi.org/10.18653/v1/D18-1425 [43] Tao Yu, Rui Zhang, Michihiro Yasunaga, Yi Chern Tan, Xi Victoria Lin, Suyi Li, Heyang Er, Irene Li, Bo Pang, Tao Chen, Emily Ji, Shreya Dixit, David Proctor, Sungrok Shim, Jonathan Kraft, Vincent Zhang, Caiming Xiong, Richard Socher, and Dragomir Radev. 2019. SParC: Cross-Domain Semantic Parsing in Context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Florence, Italy, 4511–4523. https://doi.org/10.18653/v1/P19-1443 [44] Ruiqi Zhong, Tao Yu, and Dan Klein. 2020. Semantic Evaluation for Text-toSQL with Distilled Test Suites. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Online, 396–411. https://doi.org/10.18653/v1/2020.emnlp-main.29 [45] Victor Zhong, Mike Lewis, Sida I. Wang, and Luke Zettlemoyer. 2020. Grounded Adaptation for Zero-shot Executable Semantic Parsing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Online, 6869–6882. https://doi.org/ 10.18653/v1/2020.emnlp-main.558 [46] Mark Ziemann, Yotam Eren, and Assam El-Osta. 2016. Gene Name Errors Are Widespread in the Scientific Literature. Genome Biology 17 (2016), 177.

Text-to-SQL Models Make Choices Without Saying So

These appendices add evidence and detail to the paper; they do not repeat it. Appendix A maps the data and the experiments. Appendix B shows that the choices are common: how often benchmark references meet each type, and on how many databases it occurs. Appendix C lists documented cases outside SQL in which one such choice changed a consequential number. Appendix D reports the request sets whose choice lies in a filter condition. Appendices E and F give the generator’s details, the seeds and the versions. Appendix G shows examples of every type, selected by a fixed rule. The data and code behind every number are in the reproducibility package of the repository.

A

DATA AND EXPERIMENTS

Figure S1 shows the data of the paper at three levels: the sources, the data built from them, and each experiment with its sample size. Request sets are built on the evaluation databases from generated tasks and from the choice library, which is mined from BIRD and Spider development questions; the held-out questions are not used for mining. Three subsets come from random draws made before the final selection of request sets, and the figure gives each draw with the part of it that lies in the final sets. Each experiment lists the exhibits it supports, so every figure and table of the paper can be traced to its data. The counts are generated from the same data files as the paper’s numbers.

B

Table S1: All choice types (Figure 4 shows the major ones). Library: choices of meaning in benchmark failures, open (the wording fits both interpretations) or settled (it selects one). References: open BIRD and Spider questions, by the interpretation their reference SQL takes: the usual one, or the other with its share (%), after the less common interpretations were reviewed; –: not readable from the SQL. Sets: the 204 accepted request sets, of the types in which at least 1% of open questions have a reference that takes the other interpretation. The library’s value-representation type is split by its SQL edit: comparison strictness is a strict cutoff, any other edit value matching. Library Choice type

The choice

References

Open Settled Usual Other (%) Sets

Aggregation level Rows or group averages? Calculation Which formula is meant? Quantifier “Than any”: one or all? Condition linkage Conditions on one row? Counting unit Count rows or entities? Missing values Keep or skip missing? Negation Some non-match, or none? Population scope Keep unmatched rows? Referent column Which column is meant? Relationship role Which link to follow? Strict cutoff Strict or inclusive? Ties and selection One tied row, or all? Time scope Period? Ends included? Value matching Exact or loose match?

6 1 3 1 7 8 0 0 19 2 1 7 1 1

2 244 15 (5.8) 1 613 1 (0.2) 0 8 7 (46.7) 0 260 0 (0.0) 27 2,521 237 (8.6) 3 1,630 26 (1.6) 1 272 3 (1.1) 1 12,462 5 (0.0) 6 – – 4 15 0 (0.0) 0 309 5 (1.6) 3 2,826 397 (12.3) 2 254 1 (0.4) 3 9,330 22 (0.2)

61 0 0 0 47 29 1 0 0 0 42 24 0 0

Total

57

53

204

THE CHOICES ARE COMMON

The choices in the library are not quirks of one benchmark or one database. Table S1 lists every type, how often BIRD and Spider references meet it, and how many request sets test it. Table S2 shows on how many databases, and in which sources, each type occurs. Appendix G gives examples of every type that a reader can check by hand. The library. Table S1 gives the labels and counts in the fixed example library for every type. “Open” includes both unresolved and contested records. “Settled” denotes failure records the reviewers label as settled by the wording; this label is not the word-list check used for the open-question counts of Section 6.1. Failures that differ only in output format (51 with two accepted interpretations, 10 with one) are not choices of meaning and are not in the table. Possible reference errors (27) and unplaced cases (four) are also kept apart. For Table 1, AmbiQT’s column and table ambiguity and AMBROSIA’s vagueness map to referent column, AMBROSIA’s scope to quantifier, and attachment to condition linkage. How references are counted. The References columns of Table S1 read each reference query with an SQL parser and the schema’s keys and column types, without a model, and count a question only when its wording has no cue for the choice. Ties, counting unit, missing values, population scope, “between” and value matching follow Section 6.1; calculation is a question that says percent, answered times 100 or as a fraction. Quantifier: a question that says “than any”, answered by comparing with every value (> MAX, < MIN, ALL) or with at least one (> MIN, < MAX, ANY). Negation: a negated question answered with no matching record (NOT IN, NOT EXISTS or EXCEPT over another table) or with some non-matching record (<> on the many side of a join). Aggregation level: an entity ranked

by a numeric column on the many side of a join, by one row’s value or by a sum or average per group. Condition linkage: two or more conditions on different columns of the many side of a join, on one row or on separate rows (the table joined twice by the same key, or an INTERSECT). Relationship role: two tables that share a parent and are also connected by a link table, joined through the parent or through the link table. Which column a phrase means depends on the words, not the SQL, so the referent column has no count. A lower bound. Each rule reads one SQL form of its choice, so it detects only part of the choices mined from failures: applied to the 85 library records of the twelve types, the rules find the choice in 50, including all ties and quantifier records, 27 of 34 counting records and 4 of 6 relationship records, but none of the aggregation, time scope or value matching records, whose forms (grouping by a non-key attribute, an average used as a threshold) the rules do not read. The counts are therefore a lower bound on how often the choices arise. Review of the less common interpretations. Every reference that takes the less common interpretation was then read against fixed criteria (the SQL takes that interpretation, the wording leaves the choice open, and the other interpretation could change the answer), and every verdict is listed in the package for the authors’ review. Of 1,259 such references, 540 are dropped, most often because the wording settles the choice (“list the students with the highest”, “how many different”), the join cannot repeat the counted entity, or a LIKE only escapes an apostrophe. Only this side was reviewed, so

Yichen Xu, Shipei Lin, and Chenhao Ma

Sources Databases Spider, BIRD development, LiveSQLBench: 190 tried, 152 usable for the generator. 12 evaluation databases, four per source

Models Evaluated: Haiku, Sonnet, GPT-Sol, GPT-Luna. Opus writes and reviews request sets with Sonnet, reviews Sonnet’s natural requests, and judges explanations; it is not evaluated

BIRD and Spider questions BIRD development: 481 questions on five databases to mine choices, 100 held out. Spider development: 150 to mine, 50 held out. References of all open questions, training and development

Request sets (Fig. 5) main run 215 and filter run 92 pass all checks (307); the authors keep 232; the 1% rule leaves 204 (161 without, 43 with a filter)

Choice library (Fig. 4, Table S1) 110 choices (57 open, 53 settled) in 14 types, from benchmark failures on 11 databases

Do models report the choice? (Sec. 5) Figs. 6, 8, 9 204 sets × 4 models Fig. 7 missed interpretations; also AMBROSIA Fig. 10 61 tie and counting sets Sec. 5 five samples, 55 random sets Sec. 4.1 exploration, 100 held-out BIRD questions Table S4 43 filter sets

Can benchmarks see it? (Sec. 6) Fig. 11 held-out 100 BIRD, 50 Spider × 4 models Fig. 12 failures among the mining questions Fig. 13 references of open questions, by author group Fig. 14 664 open BIRD questions, 156 change Table 4 137 BIRD questions × 4 models

AMBROSIA 60 development questions; 150 new test questions per model (90 ambiguous, 60 unambiguous)

Built from them Generated tasks on the 12 evaluation databases: 1,198 of 1,200 queries delivered; 382 tasks with a natural request

Experiments: exhibit and sample size Generator (Sec. 4.2) Fig. S4 coverage on 190 databases Table S5 1,198 of 1,200 queries Fig. S2 rules removed, 9 configurations × 2,350 attempts Fig. S3 exact vs. natural request, 382 tasks × 4 models Sec. 4.2 second writer, 200 random tasks Sec. 4.2 plan vs. direct writing, 60 tasks

Two-phase method (Sec. 7) Fig. 16, Table 5 AMBROSIA, 150 per model, Sonnet and GPT-Sol Table 5 111 random request sets Fig. 17 363 interpretations over 4 models Sec. 7.3 85 BIRD questions, Sonnet and GPT-Sol

Figure S1: Data and experiments: sources (top), data built from them (middle), and the experiments with their sample sizes (bottom). Arrows show what is built from what and what each experiment uses. the shares in the table are conservative. The same drops apply to Section 6.1.

Breadth. Table S2 counts every library choice by the benchmark whose failure revealed it, and every accepted request set by the source of its database. The library reflects where failures were diagnosed: its choices come from eleven databases, and several types were found in one benchmark only. The request sets show that the major types are not tied to those databases. Each of the five types with request sets beyond a single one occurs on at least eight of the twelve databases, and in databases from Spider, BIRD and LiveSQLBench.

Request sets. Table S1 counts the 204 accepted request sets by type, 43 of them with a choice in a filter (Appendix D). Over the 215 sets whose choice a model proposes, aggregation level contributes 63, ties 44, missing values 40, counting unit 35, calculation 15, relationship role nine, and time scope two. The remaining seven belong to the six additional labels: running-total interpretation (two), active-status filtering, comparison baseline, baseline population, baseline definition, and row-count versus quantity measurement (one each). These counts reflect the generation and selection procedure.

C

CALCULATION CHOICES WITH REAL CONSEQUENCES

Table S3 surveys documented cases, across six domains, in which one open or mistaken choice of the kind our library records changed a consequential number or decision. None is a text-to-SQL failure. Each entry was checked against the original paper or official report; we left out cases whose primary source we could not reach, such as a bank’s value-at-risk spreadsheet and a classic study of graduate admissions.

D

REQUEST SETS WITH FILTER CONDITIONS

The filter sets of Section 2.4 come from one optional rule of the generator. It adds a single condition to the first stage of a query, on the rows of that stage’s table, and builds the second interpretation at the same time. The five forms are: a date range, inclusive (>= and <=) or exclusive (> and <), with ends at the 30th and 70th percentile of a date column; a cutoff at a frequent value of a measure, >= v or > v; “without” a category, no record of the entity has it (NOT IN) or some record lacks it (IN with <>); two categories, on one record or on any records of the entity; and a per-entity count, over entities with matching rows (inner join) or over all entities (outer join, zero counts kept). The generator prefers forms it has used less often. The seed is 20261016; 161 of 180 requested queries are delivered, 121 of them have interpretations with different answers, and 92 request sets pass all checks.

Text-to-SQL Models Make Choices Without Saying So

Table S2: Top: every library choice (Table S1), by the benchmark whose failures revealed it. Bottom: every accepted request set, by the source of its database. DBs: distinct databases. Library choice type

Choices

DBs

Aggregation level Calculation Condition linkage Counting unit Missing values Negation Population scope Quantifier Referent column Relationship role Strict cutoff Ties and selection Time scope Value matching

8 2 1 34 11 1 1 3 25 6 1 10 3 4

4 2 1 4 5 1 1 1 4 2 1 6 2 3

Found in BIRD BIRD BIRD BIRD, Spider BIRD, Spider BIRD BIRD Spider BIRD, Spider BIRD, Spider BIRD BIRD, Spider BIRD BIRD, Spider

Total

110

11

BIRD, Spider

Request set type

Sets

DBs

Spider

BIRD

LiveSQL

Aggregation level Counting unit Missing values Negation Strict cutoff Ties and selection

61 47 29 1 42 24

11 10 9 1 10 10

20 16 12 1 17 5

16 5 16 0 16 8

25 26 1 0 9 11

Total

204

12

71

61

72

Algorithm S1 Build a request set around one calculation choice Require: Databases 𝐷, target depths 𝑑, required operations 𝑅 Ensure: Accepted request sets and reasons for rejected sets 1: for each requested task do 2: Sample 𝑝𝐴 ∈ P (𝐷, 𝑑, 𝑅) with the look-ahead of Eq. (3) 3: If no query is delivered, record the shortfall and continue 4: Describe 𝑝𝐴 ; write and check its request 5: Propose edits 𝑝𝐴 → 𝑝 𝐵 that express a different choice 6: Execute and describe both queries; require different answers 7: Repair failed proposals once; review both interpretations 8: end for 9: Select one choice per task at most, balancing types and databases 10: for each selected choice do 11: List what stays fixed and what changes 12: Write requests specifying A, specifying B, and leaving it open 13: Check requirements and all six request–interpretation pairs 14: Revise a failing set once and recheck 15: Keep 𝐹 if keep(𝐹 ); otherwise record the reason 16: end for

E

Results. Table S4 gives the results per model on the 43 accepted filter sets. What models raise follows the wording of the open request. Forty of the 42 cutoff requests give a value “as the cutoff” or “threshold,” with no comparison word, and the other two say “upwards of”; models return both the strict and the inclusive interpretation in 115 of the 159 cases where they compute both. The settled request is never called open. GPT-Luna returns both interpretations least often, as on the other sets.

Rejected sets. The authors reject all 17 date-range sets. Every date-range request says “between,” which SQL’s BETWEEN includes, and 254 of the 255 BIRD and Spider references whose question says “between” with no other cue include both ends. Among the BIRD and Spider references whose question says “between” with no other cue, the one that excludes the ends is a Spider reference that applies strict bounds to the start and end columns of a staff role. A BIRD reference whose evidence says BETWEEN 121 AND 130 but whose query excludes both ends is not counted, since its evidence settles the ends. Four further references use strict bounds moved one step out, or a negated range, and select the inclusive range. Of the 49 rejected sets, 17 are the date ranges, 15 are cutoffs whose wording (“passes”, “clears”, “on the high side of”) already makes the comparison strict, five are “without” requests that already mean no such record, and one is a per-entity average whose request restricts the population. The other 11 test per-entity counts with or without unmatched entities (eight) and conditions on one or any records (three), choices whose other interpretation real references almost never take (Figure 4).

QUERY GENERATION DETAILS

Algorithm S1 lists the construction of a request set, Figure S2 shows what each construction rule contributes, and Figure S3 compares the exact description with the natural request. Figure S4 shows, for each feature, the share of 152 databases whose schema allows it and the share where a generated query with it was accepted by both model reviewers; and Table S5 the queries delivered on the 12 evaluation databases. Formal statement. Let L (𝐾𝐷 ) be the queries that the catalogue 𝐾𝐷 of database 𝐷 and the type rules allow, 𝛿 (𝑝) a query’s arithmetic and step depth, and 𝑂 (𝑝) its operations. For target depths 𝑑 and required operations 𝑅 the generator samples from P (𝐷, 𝑑, 𝑅) = {𝑝 ∈ L (𝐾𝐷 ) : 𝛿 (𝑝) = 𝑑, 𝑅 ⊆ 𝑂 (𝑝), 𝐼𝐷 (𝑝) = 1}, (2) where 𝐼𝐷 (𝑝) = 1 means that all data checks pass. It keeps a partial query 𝑧 only if some completion can still reach the target, can(𝑧, 𝑑, 𝑅) = 1{∃(𝑑, 𝑈 ) ∈ Reach(𝑧) : 𝑅 ⊆ 𝑈 },

(3)

where Reach(𝑧) lists the depths and operation sets still possible. The look-ahead is exact for the rules, and no query is drawn twice. For a request set 𝐹 with specific requests 𝑞𝐴 , 𝑞𝐵 and open request 𝑞 ◦ , let 𝐻𝑖(𝑟𝑗 ) = 1 when reviewer 𝑟 accepts request 𝑖 for interpretation 𝑗, 𝑓𝑟 = 1 when 𝑟 finds no changed or missing requirement, and 𝑐 (𝐹 ) = 1 when all code checks pass. The set is kept when keep(𝐹 ) = 𝑐 (𝐹 ) ∧

Ûh 𝑟

(𝑟 ) 𝑓𝑟 ∧ 𝐻 {𝐴,𝐵 = 𝐼2 },{𝐴,𝐵 }

i Üh (𝑟 ) (𝑟 ) ∧ 𝐻 ◦𝐴 ∧ 𝐻 ◦𝐵 . 𝑟

i (4)

Yichen Xu, Shipei Lin, and Chenhao Ma

Table S3: Documented cases where one calculation choice changed a consequential number or decision, by domain, with the closest choice type in our library. Domain

Case

Choice and consequence

Library type

Economics and policy

Public debt and growth [11]

Growth was averaged within each country before averaging across countries, so one year of New Zealand data weighed as much as decades of another country. With corrected data and weighting, high-debt countries grew 2.2% a year, not −0.1%.

aggregation level

Health care

Risk algorithm for extra care [25]

“Need” was computed as health-care cost. The algorithm looked accurate by standard measures, yet fixing the proxy would raise the share of Black patients receiving extra help from 17.7% to 46.5%.

referent column

Health care

Hospital mortality rates [33]

Four methods, given the same 2.5 million discharges, included 28–95% of them. 12 of 28 hospitals with higher-than-expected mortality by one method had lower-than-expected mortality by another.

population scope

Public health

1995 UK contraceptive pill warning [8]

A warning of higher thrombosis risk for two pill types was followed by pill use among under-16s falling from 40% to 27%, and by estimated NHS costs of £21 million for maternity care and £46 million for abortions. The author concludes that risk should be more carefully presented.

how a quantity is stated (not in our library)

Social science

Referees and skin tone [34]

29 teams analysed the same data. Their odds ratios ranged from 0.89 to 2.93; 20 teams found a significant effect and 9 did not, using 21 combinations of covariates.

several analysis choices

Science

Brain imaging [4]

70 teams analysed one dataset to test the same 9 hypotheses. No two chose the same workflow, and the results of the hypothesis tests varied substantially.

several analysis choices

Science

Gene names in spreadsheets [46]

Spreadsheet software converts gene names to dates and numbers by default; about one fifth of papers with supplementary gene lists in spreadsheets contain such conversions.

value matching

Engineering

Mars Climate Orbiter [22]

Thruster data in English units were used where the trajectory software expected metric units; the spacecraft was lost.

value matching

Engineering

Ariane 5 flight 501 [20]

A 64-bit floating-point value was converted to a 16-bit signed integer that could not hold it; the launcher broke up about 40 seconds after lift-off.

value matching

Valid, on target, sound Valid, on target, not sound Duplicate

Table S4: Filter request sets (43 accepted), per model: both specific requests right (with exploration); among those, open requests answered with both interpretations; open requests that name the choice (Opus judge); specific requests right (single call); missed interpretations the model computes when asked, of all missed. Model Haiku Sonnet GPT-Sol GPT-Luna

Both right

Both returned

Named

Right

Missed, computable

42 41 42 38

33 36 32 14

40 38 35 25

40 42 40 41

9 / 10 5/6 10 / 10 24 / 29

All mechanisms

100%

− deduplication

92%

− step data checks

89%

− column profile

31%

− look-ahead

3%

− “must vary”

92%

− restatement rules

60%

− type rules

28%

None (naive)

0.1% 0

Table S5: Program construction on 12 databases, with no model calls after catalogue preparation. Every delivered SQL is distinct, has the requested depth, and passes both integrity checks.

Source

DBs

Delivered

Constructions

Seconds

Spider BIRD LiveSQLBench

4 4 4

398/400 400/400 400/400

289 270 309

127 274 37

Total

12

1198/1200

868

437

Source-level times are rounded individually; the total is rounded from the unrounded sum. Separate task review delivers 384/400 tasks, and language generation supplies 382/384 requests.

Off target depth Invalid on the data Not built

20 40 60 80 100 % of 2,350 attempts per configuration

Figure S2: Generation outcomes with each construction rule removed (2,350 attempts). The labels give the share valid, on target and sound.

F

SEEDS, VERSIONS AND SELECTION RULES

Seeds, versions and reruns. The grammar is compose-v3.4. Task requests, Spider splitting, request set selection, explanation allocation, and bootstrap resampling use seed 20261003; BIRD holdout selection uses 20261001; the paired wording comparison uses 20261004. Coverage uses seed 1 and depth-control runs seeds 1–3. Python was 3.11.5 and SQLite 3.41.2. During the experiments, the Claude command-line client changed from 2.1.287 to 2.1.289 and the Codex client from 0.147 to 0.160. Temporary usage-limit outputs were removed and retried rather than treated as SQL answers. A

Text-to-SQL Models Make Choices Without Saying So

Exact description

Natural request

+2.6

−2.4

−2.1

+9.9

90 80 70 60

Errors, exact description (%)

Tasks right (%)

100

shown in the main text. These selection rules support traceability; illustrative examples are not frequency estimates.

Errors by step depth

Overall (n = 382)

Haiku Sonnet GPT- GPTSol Luna

GPT-Luna

30 20

Haiku 10 0

Sonnet, GPT-Sol 3

4 5 6 Step depth

Figure S3: Exact description versus natural request on 382 generated tasks. Left: accuracy; numbers give the difference in points. Right: errors from the exact description by step depth. Possible by the schema Generated and accepted by both reviewers Step depth ≥ 5 Arithmetic depth ≥ 2

Share of total Baseline comparison Aggregation Grouping Joins Top-k within group Top-k rows

Ratio of two measures Difference of two measures 0

25 50 75 % of 152 databases

100

Figure S4: Features the generator can build, as a share of 152 databases: possible by the schema, and generated and accepted.

final query that failed under concurrent execution was rerun alone: ten such answers then matched their reference (nine GPT-Luna, one GPT-Sol). Token counts are unavailable for the explanation runs, so cost is reported as calls per request. Rewrite comparison. The comparison of the original specific request (a) with the other texts was fixed before the run. The comparisons with the edited request (c) were added after an interim look at half of the interpretations. Response selection. The explanation examples are the first saved responses displaying each tested pattern in file order. The worked aggregation example uses the smallest case-identifier hash among eligible request sets. Library examples use the first eligible record per type at high confidence, or medium confidence if none is high, with fixed benchmark and task ordering. Generated examples outside library labels enumerate retained cases before selecting the four

G

EXAMPLES OF EACH CHOICE TYPE

Tables S6 and S7 show examples of each type, selected by a fixed rule rather than by hand. Library choices are taken from different databases, open choices before choices the wording settles, one from each benchmark when the type has both, shortest question first. Request sets are taken from different databases, one per source in turn, shortest open request first. Each example’s question or request is given as the model received it. Answers says how the two reference queries’ results differ on the database. Models gives each model’s answer to the open request in one call with explanation, in the order Haiku, Sonnet, GPT-Sol, GPT-Luna: B, both interpretations returned; N, one interpretation, with the choice named; S, one interpretation given silently, although the model computes each interpretation when asked; a dash, one interpretation, and the model does not compute both. The package holds every choice and every set, with its SQL.

Yichen Xu, Shipei Lin, and Chenhao Ma

Table S6: Selected library choices, up to 3 per type. Source

Question

Aggregation level (3 of 8 shown) BIRD, california_ Which active district has the highest average score in Reading? schools BIRD, financial Who are the account holder identification numbers whose who have transactions on the credit card with the amount is less than the average, in 1998? Evidence: Operation = ’VYBER KARTOU’ refers to credit card withdrawal BIRD, thrombosis_ For all patient born in 1982, state if their albumin is within normal range. Evidence: prediction Year(Birthday) = ’1982’; albumin is within normal range refers to ALB between 3.5 and 5.5

The choice

Whether to rank districts by one school’s AvgScrRead value or by open the average of AvgScrRead across each district’s schools Which transactions the ’average’ amount threshold is computed over open

Whether the normal/abnormal status is given once per laboratory open record or once per patient born in 1982, since each patient can have many Laboratory rows

Calculation (2 of 2 shown) BIRD, financial Provide the IDs and age of the client with high level credit card, which is eligible for loans. How to compute the client’s age from birth_date: subtract the birth Evidence: the credit card is high-level refers to card.type = ’gold’; eligible for loans refers to year from the current year, or count completed years of life. disp.type = ’OWNER’ BIRD, thrombosis_ For all patients who are older than 55 years old, what is the percentage of female who has Which population the percentage is taken over (the denominator) prediction abnormal prothrombin time (PT)? Evidence: older than 55 years old = SUBTRACT(year(current_timestamp), year(Birthday)) > 55; abnormal prothrombin time (PT) refers to PT > = 14; percentage = DIVIDE(SUM(PT > = 14 AND SEX = ’F’), SUM(PT > = 14)) * 100; female refers to sex = ’F’; Condition linkage (1 of 1 shown) BIRD, toxicology Calculate the percentage of carcinogenic molecules with triple bonded Hidrogen atoms. Evidence: hydrogen refers to element = ’h’; label = ’+’ mean molecules are carcinogenic; triple bond refers to bond_type = ’#’; percentage = DIVIDE(SUM(label = ’+’), COUNT(molecule_id)) * 100.0 where element = ’h’ AND bond_type = ’#’;

Spider, cre_Doc_ Template_Mgt

Show all template ids and number of documents using each template.

Spider, network_1

Show the student IDs and numbers of friends corresponding to each.

Negation (1 of 1 shown) BIRD, toxicology

What are the countries that have greater surface area than any country in Europe?

Referent column (3 of 25 shown) BIRD, california_ Please specify all of the schools and their related mailing zip codes that are under Avetik schools Atoian’s administration. BIRD, financial

What year was account owner number 130 born?

Spider, car_1

What are the makers and models?

Relationship role (2 of 6 shown) BIRD, toxicology Which molecule consisted of Sulphur atom with double bond? Evidence: sulphur refers to element - ’s’; double bond refers to bond_type = ’ = ’; Spider, network_1

Strict cutoff (1 of 1 shown)

settles it

open

open

open

How to handle the top AvgScrMath row in satscores when it has no open matching record in schools: drop it and take the best-scoring school that does have a record, or keep it as the answer even though its phone is missing. Whether templates that no document uses should appear in the open output with a count of 0, or only templates that at least one document uses should be listed Whether students who have no rows in Friend should be listed with open a count of 0, or left out.

What percentage of carcinogenic-type molecules does not contain fluorine? Evidence: label = What it means for a carcinogenic molecule to ’not contain fluorine’: settles it ’+’ mean molecules are carcinogenic; contain fluorine refers to element = ’f’; percentage = the molecule has at least one atom that is not fluorine, or the DIVIDE(SUM(element = ’f’) * 100, COUNT(molecule_id)) where label = ’+’; Should consider molecule has no fluorine atom at all. the distinct atoms when counting;

Population scope (1 of 1 shown) BIRD, thrombosis_ How many patients with an Ig G higher than normal? Evidence: Ig G higher than normal prediction refers to IGG >= 2000; Should consider DISTINCT in the final result;

Quantifier (1 of 3 shown) Spider, world_1

open

How a hydrogen atom is tied to a triple bond: either the hydrogen is open one of the two atoms in the ’#’ bond (linked through connected), or the hydrogen and some ’#’ bond just appear in the same molecule (linked only through molecule_id).

Counting unit (3 of 34 shown) BIRD, thrombosis_ What is the average age of the male patient with high cholesterol? Evidence: average age = Whether the average is taken once per patient or once per matching prediction DIVIDE(SUM(SUBTRACT(YEAR(NOW()), YEAR(birthday))), COUNT(ID)); male patient lab record, since a patient can have several Laboratory rows with refers to sex = ’M’; high cholesterol refers to T-CHO > = 250; T-CHO >= 250 BIRD, toxicology What is the least common element of all carcinogenic molecules? Evidence: label = ’+’ mean What unit ’least common’ is counted in: the number of carcinogenic molecules are carcinogenic molecules that contain the element, or the number of atoms of that element across all carcinogenic molecules Spider, student_ How many different degrees are offered? What counts as one degree: a distinct degree name transcripts_tracking (degree_summary_name) or each degree program row in Degree_Programs Missing values (3 of 11 shown) BIRD, california_ What is the phone number of the school that has the highest average score in Math? schools

Wording

What is the lowest grade of students who do not have any friends?

Whether the patient count only includes patients who also have records in the Examination table (because of the inner join to Examination), or every patient with any lab IgG result of at least 2000

settles it

Whether ’greater than any country in Europe’ means larger than at open least one European country (compare to the smallest European area) or larger than every European country (compare to the largest European area)

open Which administrator slots count as being ’under Avetik Atoian’s administration’: only the primary administrator columns (AdmFName1/AdmLName1), or any of the three administrator slots. Which ID the number 130 in ’account owner number 130’ refers to: open the account ID or the client ID Which column supplies ’makers’: the integer foreign key open model_list.Maker, or the maker name car_makers.Maker reached by joining on that key

Does the sulfur atom itself have to be part of the double bond, or is it open enough that the molecule has both a sulfur atom and a double bond somewhere? Whether a student who appears in Friend only as friend_id open (someone else lists them as a friend) counts as having a friend

Text-to-SQL Models Make Choices Without Saying So

Source

Question

BIRD, thrombosis_ prediction

Provide the ID and age of patient with lactate dehydrogenase (LDH) between 100-300 index Whether the LDH range endpoints 600 and 800 are included (strict above the normal range. Evidence: age refers to SUBTRACT(year(current_timestamp), vs. inclusive bounds) year(Birthday)); lactate dehydrogenase (LDH) between 100-300 index above the normal range refers to LDH between 600 and 800;

The choice

Wording open

Ties and selection (3 of 10 shown) BIRD, superhero Give the hero ID of superhero with the lowest attribute value. Evidence: lowest attribute value refers to MIN(attribute_value); BIRD, thrombosis_ prediction Spider, world_1

How to handle ties at the minimum attribute_value: return every open tied hero, or return just one hero as the singular ’the hero ID’ suggests. How to handle several lab records tied at the maximum in-range IgA open For the patient who has the highest Ig A within the normal range, what is his or her diagnosis? Evidence: highest Ig A within the normal range refers to MAX(IGA BETWEEN 80 value (IGA = 500): return one row or every tied patient’s diagnosis. AND 500); Give the name of the nation that uses the greatest amount of languages. Several countries tie for the most languages (12). The question does settles it not say which of them to return, so the answer depends on how the tie is broken.

Time scope (2 of 3 shown) BIRD, financial For the client who applied the biggest loan, what was his/her first amount of transaction after opened the account? BIRD, thrombosis_ For all patients with normal uric acid (UA), what is the average UA index based on their prediction latest laboratory examination result? Evidence: uric acid (UA) with normal range refers to UA < 8.0 and SEX = ’M’ or UA < 6.5 and SEX = ’F’; average UA index refers to AVG(UA)

Whether a transaction on the account-opening date counts as ’after open opened the account’ (inclusive vs. strictly later boundary) Which date counts as the ’latest laboratory examination’: the single settles it latest date in the whole Laboratory table, or each patient’s own latest test date

Value matching (3 of 4 shown) BIRD, financial How much, in total, did client number 617 pay for all of the transactions in 1998? BIRD, thrombosis_ prediction Spider, car_1

Which 1998 transactions count as the client having ’paid’: every transaction on the client’s accounts, or only outgoing ones (debits). Please list the disease names of the patients that have a proteinuria level higher than normal. Whether the threshold U-PRO >= 30 compares the TEXT column Evidence: disease names refers to Diagnosis; proteinuria level higher than normal refers to U-PRO as text (string order) or as numbers U-PRO > = 30; Which model saves the most gasoline? That is to say, have the maximum miles per gallon. Whether the TEXT-typed MPG column is ranked by its stored string value or by its numeric value when picking the maximum

open settles it

settles it

Table S7: Selected accepted request sets, up to 4 per type. Set

Answers

Models

Aggregation level (4 of 61 shown) chinook_1 Which media types (by name, counting only tracks that have a media type) have 1adc54e3cf a bytes-per-millisecond rate higher than the simple average of that rate across all media types? For each, show the media type name, its rate, and how much it exceeds that average, unrounded.

value 181.7 vs 179.5

-BBN

formula_1 c759056a31

value 0.039 vs -0.03776

S--S

value 3.281 vs 3.334

SSSS

value 24.9 vs 25.03

SBBN

Counting unit (4 of 47 shown) sakila_1 For each actor, show their actor ID, how many films they’re credited in, that fc16192583 count as a fraction of the total, and how far that fraction is above or below the average fraction across actors. Return the top 10 by that difference, highest first, unrounded.

value 0.001591 vs 0.008716

-N--

formula_1 c1a8ce4d18

16 vs 17 rows

SS--

value 297 vs 30

SBSS

value 242 vs 241

BBBS

thrombosis_ prediction 7fd142e2d9

alien cde68b2348

student_club 6b346d7be2

insider e6f24d2438

Open request

Interpretations

A: Each media type’s rate is the mean of its tracks’ individual Bytes/Milliseconds ratios; list the media types whose rate is above the unweighted mean of all media types’ rates, with the rate and how far above it is. B: Each media type’s rate is its total Bytes divided by its total Milliseconds; list the media types whose rate is above the unweighted mean of all media types’ rates, with the rate and how far above it is. For each circuit country, what are the average constructor standings points and A: A country’s averages are the mean of per-race averages, so each wins (the season-to-date figures) at races held there? Also, how far is each race counts equally. country’s average wins above or below the average of the country figures? B: A country’s averages are the plain mean over all of its standings records pooled together. By patient sex, what are the average aCL IgG and ANA values in the examination A: Average aCL IgG and ANA over all matched examination data (only examinations linked to a patient, missing values ignored)? Also give records within each sex. Each examination counts once. each sex’s average ANA as a ratio of the mean of the per-sex ANA averages, B: Average each patient’s examinations into per-patient values unrounded. first. Then average those values within each sex, so each patient counts once. For observatory stations whose average telescope bandwidth usage A: Compare each station’s average bandwidth usage with the (bandusagepct) is above average, what is their mean bandwidth usage, and how unweighted mean of the station averages. far above average are they on average? Ignore missing usage values, and give B: Compare each station’s average bandwidth usage with the both figures unrounded in a single row. mean bandwidth usage over all individual telescopes.

A: Each actor’s film count is divided by the total number of casting credits in film_actor, which gives the actor’s share of all credits. B: Each actor’s film count is divided by the number of distinct films that have a cast in film_actor, which gives the fraction of those films the actor appears in. Which circuit cities have an above-average share of qualifying? For each, show A: Each city’s count is its number of driver qualifying entries, and the city, its qualifying count, that count as a share of the total across all cities, and its share is its share of all qualifying entries. how far its share is above the average city share (all unrounded). B: Each city’s count is its number of distinct races with qualifying data (qualifying sessions), and its share is its share of all such sessions. Taking attendance entries that match an event (others left out), how many A: A status’s attendance is the total number of member-event attendees does each event status have, and what share of the total across all attendance entries for events with that status. statuses is that? Show the status, its number of attendees, and the share, without B: A status’s attendance is the number of different members who rounding. attended any event with that status. For each trader type, how many analysed trade records are there in the advanced A: For each trader type, count the advanced behaviour analysis behaviour data (only where the trade and its trader can be found), and what share records that join to a transaction and a trader. Report that count of the overall total does each type make up? Keep the share unrounded. and its share of the total count over all trader types. B: For each trader type, count the distinct transactions that have at least one advanced behaviour analysis and a matching trader. Report that count and its share of the total count over all trader types.

Missing values (4 of 29 shown)

Yichen Xu, Shipei Lin, and Chenhao Ma

Set

Open request

Interpretations

Answers

Models

baseball_1 f74c2bad9d

For players who received award votes (matched to player by player_id), what is the average points_won for each death country (ignoring missing points)? Show each death country, its average, and how far it is from the simple average of those per-country averages, without rounding.

8 vs 7 rows

NN-N

formula_1 af91f25689

Which 10 drivers have the most qualifying entries compared with the average driver? For each, show the driver ID, how many qualifying records they have, and that number divided by the average number of qualifying records per driver (unrounded), highest ratio first.

value 4.112 vs 22.63

SBBB

student_club 55be105b98

Group members by their major (link_to_major) and show each major id, how many members it has, and that count’s share of the total. List the top 10 by share, highest first, with no rounding.

10 rows; 1 differ

BBBN

crypto b61f4a52bd

For each order in orderexecutions.ordersmark (executions with no order reference grouped together), what’s the average size (fillcount) of the fills it got? Keep the orders whose average beats the average across all orders, and note by how much. Then, among those, keep the ones that also beat that group’s own average, and note that margin too. Give me one summary row with three values: the average per-order fill size of the final orders, their average margin over the first benchmark, and their average margin over the second. Leave everything unrounded.

A: Average the points won per death country, with a blank death country kept as its own group. Compare each group’s average with the plain average of all group averages, the blank group included. B: Average the points won per death country, leaving out players with no recorded death country. Compare each country’s average with the plain average of the remaining countries’ averages. A: The average number of qualifying records per driver is computed only over drivers who appear in the qualifying table. B: The average number of qualifying records per driver is computed over every driver in the drivers table, with drivers who never qualified counting as zero. A: Members without a major form their own group, and they are included in the total used as the denominator. B: Members without a major are left out of both the groups and the total used as the denominator. A: Zero-quantity executions count as fills of 0 in each order’s average fill quantity. B: Zero-quantity executions are not fills and are left out of each order’s average fill quantity.

value 4.024 vs 4.021

SSSS

Negation (1 of 1 shown) customers_and_ Ignoring transactions with no invoice number, which invoice numbers have invoices transactions without the type ’Payment’ (exact text)? Show each invoice number 2e529bb193 with the total number of transactions recorded for it. Sort by that count from (filter) highest to lowest and give the top 10.

A: Invoices with no Payment transaction at all. For each one, 4 vs 7 rows count its transactions and show the top 10 by count. B: Invoices with at least one transaction that is not a Payment. For each one, count all its transactions (Payments included) and show the top 10 by count.

Strict cutoff (4 of 42 shown) chinook_1 Using 3.98 as the cutoff on the invoice total (Invoice.Total) to keep only the A: Invoices whose total equals 3.98 are included in the average. 06934ae88f higher-value invoices, what is their average total, as a single unrounded value? B: Only invoices whose total is strictly greater than 3.98 are (filter) Ignore missing totals, and give a missing result if no invoices qualify. included in the average. financial Applying a cutoff of 2802.0 to the debited amount (order.amount), what is the A: Orders of exactly 2802.0 are included. 8a55588ad2 average debited amount of the large standing orders, unrounded? B: Only orders above 2802.0 are included. (filter) formula_1 Among driver standings entries where the driver has upwards of 1 point so far A: Average the season-to-date win counts over standings entries 57277c47b4 that season, what is the average season-to-date win count? Ignore missing values whose season-to-date points are 1 or more. (filter) and don’t round. B: Average the season-to-date win counts over standings entries whose season-to-date points are more than 1. Using 0.149 as the cutoff, how many advanced behaviour records have high peer A: Peer correlation values equal to 0.149 are counted (at least insider correlation (peercorr)? Give a single count. 0.149). 8ed79619ea B: Only peer correlation values strictly greater than 0.149 are (filter) counted (more than 0.149). Ties and selection (4 of 24 shown) sakila_1 What are the top 10 films by number of copies in our inventory? For each, show b77a784ee8 the film_id, how many copies we have, and what share of all inventory copies that is (copies divided by the total number of copies, unrounded), with the highest share first.

financial 53afca1d49

What are the top 10 branch districts (account.district_id) by share of all accounts? For each, show the district id, its number of accounts and its share of all accounts, without rounding, listed from the highest share down.

student_club 5c048dd871

Which are the top 10 members by share of total event attendance? Match attendance to members by member id, count attendance per member id and position, and show the member id, position, attendance count and share of all attendance, from highest share to lowest.

alien ee6b268047

What are the top 10 telescopes by share of detected signals? For each telescope reference (telescref), show how many signals it detected and what fraction of all signals that is (unrounded), from the largest share down.

A: List the top 10 films by share of inventory copies, returning exactly 10 rows and choosing arbitrarily among films tied at the cut. B: List the top films by share of inventory copies, keeping every film whose copy count reaches the 10th-highest count, so ties at the cut are all included. A: Rank districts by their share of accounts and return exactly the first 10 rows, dropping tied districts arbitrarily at the cut. B: Rank districts by their share of accounts and return every district whose share reaches the 10th-highest share, including all ties at the cut. A: Rank member/position groups by their share of all attendance and return exactly 10 rows, cutting arbitrarily through any tie at 10th place. B: Rank member/position groups by their share of all attendance and return the top 10 plus every row tied with the 10th row’s share. A: Rank telescopes by their share of all signals and return exactly 10 rows, choosing arbitrarily among telescopes tied at the 10th place. B: Rank telescopes by their share of all signals and return every telescope whose share is at least the 10th-highest share, including all ties at the cut.

SSNS

1 rows; 1 differ

BBBS

1 rows; 1 differ

BBBB

1 rows; 1 differ

BBBS

1 rows; 1 differ

BBBN

10 vs 72 rows

-N--

10 vs 12 rows

SNSS

10 vs 12 rows

SNSS

10 vs 51 rows

-NS-

Related documents

• CVE-2026-94082 — HIGH 7.6#1159225
• Task Competence Is Not Instruction Following: Evaluating Instruction-Conflicting Behavior in Small Language Models#1342563
• Benchmarking Text-to-SQL under Role-Based Access Control#1062968
• Register Bias in Complexity-Based Large Language Model Routing#1011276
• TriageRA-CCF: Source-Side Clinical Confidence and Coverage Signals for Adaptive Rank Budgeting in Medical LLMs#1131595