Same Data, Different Schemas: Robustness of LLM-based Text-to-SQL Nitin Kanchinadam, Aditya Menachery, Amol Deshpande
arXiv:2605.25838v1 [cs.DB] 25 May 2026
[email protected],[email protected],[email protected] University of Maryland, College Park, USA
Abstract
1
Large language models (LLMs) consistently achieve strong results on text-to-SQL benchmarks, but their robustness to schema variations remains poorly understood. Recent work suggests that the schema structure matters, but does not provide a clear and systematic way to evaluate model behavior when different schemas represent the same underlying data. We address this problem by presenting a framework to evaluate and benchmark text-to-SQL techniques over equivalent relational schemas generated from a common E/R model. By varying the “shredding” choices used to translate the conceptual design into relations, we create multiple schema variants that differ structurally while preserving the same underlying semantics. This gives us a controlled setting in which the natural language questions and data remain fixed, and only the schema changes. We use this framework to evaluate four leading LLMs on the same questions across multiple schema variants (for two separate domains), and summarize consistency patterns using pairwise comparison heatmaps. Our results show that schema structure significantly affects LLM behavior: across conceptually equivalent schemas, models often produce SQL queries with very different answers. We also find that providing additional context (specifically, the original E/R specification) improves the performance somewhat, but does not fully ameliorate the inconsistencies. In addition to demonstrating that the current text-to-SQL evaluations miss an important notion of robustness, our framework provides a way to generate a large number of synthetic datasets that can be used to train new models, and suggests a mechanism to make text-to-SQL more robust by generating additional candidate plans for a given natural language query through systematic schema variations.
Text-to-SQL (NL2SQL) systems aim to bridge the gap between natural language and structured query languages, and there is a large body of work on developing such systems [1, 5, 7, 9, 12, 14, 15, 17– 19, 22]. Recent advances in large language models (LLMs) have led to major improvements on standard benchmarks like Spider [23] and BIRD [2]. On Spider 2.0, leading systems now report test accuracy in the mid-80s to low-90s, and prompt-based systems using strong LLMs have reached 96.7% on the test set1 . At the same time, anecdotal evidence from real-world deployments indicates a large gap between success on standard benchmarks and performance in more realistic settings [16], leading to continued development of new techniques [7, 21] as well as benchmarks (e.g., BIRD-Interact [8]). A common theme in the recent work is that schema structure matters. Many successful systems explicitly model schema structure, perform schema linking, or both, and prior work has argued that schema linking is one of the main bottlenecks in Text-to-SQL [11]. More recent work has also shown that current models can be sensitive to changes in schema organization. Li et al., show that the same question may require substantially different SQL when the schema structure changes, and that model performance drops sharply on schema-altered versions of existing benchmarks [13]. Fürst et al., study robustness across three different data models for the same football domain and show that performance can vary significantly across alternative schemas built over the same underlying data [6]. Other recent work has looked at robustness under schema evolution and at the effect of normalization choices, again showing that schema design can have a strong effect on LLM-based SQL generation [10, 24]. These results point to an important gap in current evaluation. Standard text-to-SQL benchmarks mostly test cross-domain generalization: can a model answer questions over databases and schemas it has not seen before? However, in many real applications, the same underlying data can be represented by different relational schemas. This can happen because of normalization and denormalization choices, because relationships are represented differently, or because an E/R design is translated into relations in different ways. This is especially common in domains with inheritance or subtype structure, where there are multiple reasonable shredding choices. From the perspective of the user, however, the questions or the data have not changed, only the logical representation of the data in the database, i.e., the schema, has changed. This leads to the question we study in this paper: Do LLM-based Text-to-SQL systems produce consistent and correct SQL when the
Keywords Text-to-SQL; NL2SQL; E/R Model; Schema Equivalence ACM Reference Format: Nitin Kanchinadam, Aditya Menachery, Amol Deshpande. 2026. Same Data, Different Schemas: Robustness of LLM-based Text-to-SQL. In Nineth International Workshop on Exploiting Artificial Intelligence Techniques for Data Management (aiDM ’26), May 31-June 05, 2026, Bengaluru, India. ACM, New York, NY, USA, 12 pages. https://doi.org/10.1145/3814940.3815329
This work is licensed under a Creative Commons Attribution 4.0 International License. aiDM ’26, Bengaluru, India © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2719-1/2026/05 https://doi.org/10.1145/3814940.3815329
Introduction
1 https://spider2-sql.github.io, Retrieved May 8, 2026
aiDM ’26, May 31-June 05, 2026, Bengaluru, India
Nitin Kanchinadam, Aditya Menachery, Amol Deshpande
Relational Schemas E/R Diagram
Equivalent RDBMS Instances E/R Data
Cross-Schema Consistency Analysis
Data Generation Parameters NL Questions Figure 1: Overall benchmark generation and evaluation pipeline. Most of the steps, including initial E/R diagram, can themselves be fully automated using LLMs. schema changes, even when the question and the underlying data remain the same? We study this question through the lens of alternative E/R-torelational shredding choices. Starting from a common E/R model, we generate multiple relational schemas that are different at the logical level but equivalent at the conceptual level. We then populate these schemas and evaluate text-to-SQL systems on the same natural-language questions across the resulting schema variants2 . This gives us a systematic way to measure robustness to schema variation that arises from database design choices rather than from changes in domain or user intent. Our work builds on recent work that studies the space of alternative relational representations for a common conceptual design [4]. That setting is a natural fit for the robustness question we care about here, since it lets us generate many equivalent schema variants in a principled way. It also exposes a challenge that is easy to overlook in this setting: ground truth. When two schemas differ, the correct SQL queries may also differ substantially in surface form even when they express the same intent. Hence, we focus on comparing the execution behavior across schema variants, i.e., we compare the results of the queries across the databases. We note that this provides, at best, a lower bound on the inconsistency rates we can observe; an interesting direction of further research is to try to construct databases that can help better identify mismatches across the queries (analogous to work on killing SQL query mutants [20]). The main contributions of this paper are as follows: • We present a framework for generating and populating many equivalent relational schemas from a common E/R model by varying shredding choices. • We use this framework to build text-to-SQL evaluation settings in which the natural-language questions remain fixed while the schema varies systematically. • We evaluate several leading LLMs on these schema variants and show that they exhibit substantial inconsistencies (even 2 The data generation scripts and datasets are available at: https://github.com/umddb/
text2sql-equivalent-schemas.
if the E/R model is provided) across schemas that should, in principle, be equivalent. More broadly, our results suggest that current text-to-SQL evaluations do not fully capture an important notion of robustness: invariance to alternative relational representations of the same underlying domain. We believe this is a natural and practically important direction for evaluation, especially as LLM-based systems are increasingly used by non-experts who may not be sufficiently familiar with SQL. Our work also suggests that the existing text-to-SQL systems could be made more robust by systematically incorporating schema variations into the process. For instance, one could create a few different equivalent databases on the fly with different schemas (using an existing E/R diagram or reverse-engineering one [3]), and generate SQL queries for a given natural language prompt simultaneously across those databases. Differences across the results of those queries could be used to flag potential problems with the SQL queries that are generated. We are planning to investigate this and similar approaches in our future work.
2
Dataset Generation Framework
In our framework, we treat an E/R specification as the semantic source of truth, and the relational schema as only one possible implementation of that specification. Starting from a single E/R diagram, we generate multiple relational schemas that are conceptually equivalent, populate them with instances derived from the same conceptual data, and then evaluate the same natural-language questions over all schema variants. This viewpoint is closely aligned with the broader argument in prior work that the E/R model provides a more stable abstraction layer than the relational schema, and that choices such as how to represent inheritance, relationships, and multi-valued attributes should be viewed as implementation decisions rather than user-visible semantics [4]. Figure 1 shows the overall workflow, where we start with an E/R diagram and a small number of generation parameters, such as the desired scale of the data and distributions governing relationship
Same Data, Different Schemas: Robustness of LLM-based Text-to-SQL
Creator bio verified
aiDM ’26, May 31-June 05, 2026, Bengaluru, India
Account
Content
Channel
username email created_at
title created_at duration quality_opts[]
name description created_at
Consumer
LiveContent
preferences age watch_history _timestamps[]
scheduled_start is_live
RecordedContent Tag
file_size_mb resolution chapter_markers[]
name aliases[]
Streamer
Subscriber
LiveStream
Clip
platform avg_viewers
subscription_tier active notification_prefs[]
peak_viewers chat_enabled
start_offset end_offset
Figure 2: Social Media E/R diagram used as a running example. We omit relationship names for brevity. cardinalities. From this, we first generate a conceptual instance of the E/R model, containing entity and relationship instances. We then apply a collection of shredding choices to translate the conceptual instance into multiple relational schemas and corresponding database instances. In parallel, we generate natural-language questions whose meaning is tied to the E/R model rather than to any one schema, and evaluate the same questions against each of the generated databases. The initial steps in this pipeline, specifically the generation of the E/R diagram and the natural-language questions, may be automated to generate new benchmarks. However, one could also read the schema from an existing database, and create schema variants from that (either through creating an E/R diagram first, or through applying rules directly to the relational schema).
2.1
Running Example: Social Media Domain
We use a social-media application as a running example throughout this section (Figure 2). The schema contains two nontrivial inheritance hierarchies, several multi-valued attributes, and a mix of many-to-one and many-to-many relationships, making it a good domain for illustrating the space of equivalent relational encodings. Specifically, one inheritance hierarchy contains an Account entity, that is specialized into CreatorAccount and ConsumerAccount. Those two are further specialized into Streamer and Subscriber respectively, with appropriate sets of attributes as shown. The other hierarchy is around Content, specialized into LiveContent, LiveStream, RecordedContent, and Clip. The schema also includes standalone entities such as Channel and Tag. Several attributes are multi-valued, including Tag.aliases, Content.quality_options, RecordedContent.chapter_markers, ConsumerAccount.watch_history_timestamps, and Subscriber.notification_preferences. The relationship structure is equally rich. Every piece of content has a producer, so there is a many-to-one relationship from Content to CreatorAccount. Every live stream has a host, giving a manyto-one relationship from LiveStream to Streamer. Clips may be
derived from live streams, giving another many-to-one relationship. The schema also contains many-to-many relationships indicating which content is published on which channels, which tags are attached to which content, which creators own which channels, which consumers viewed which content, which subscribers follow which creators, and which accounts shared which clips.
2.2
Generating Equivalent Relational Schemas
Next, we discuss how we explore the family of equivalent relational schemas by systematically varying a small set of local design decisions. One major source of variation is the representation of inheritance hierarchies. In the social-media example, both Account and Content form multi-level subtype hierarchies. A schema designer could store all members of a hierarchy in a single relation using a discriminator (type) attribute, often called single-table inheritance (STI) pattern. Alternatively, the designer could place the common attributes in a root table and store subtype-specific attributes in separate subclass tables, corresponding to class-table inheritance (CTI). A third possibility is to create separate concrete tables for the leaf classes and duplicate inherited fields into those tables (called concrete table inheritance (CONCRETE)). These are large structural changes but preserve the semantics. A question about subscribers or streamers may require a simple filter predicate under one schema and a chain of joins under another. To make this concrete, consider the Account hierarchy. Under a single-table encoding (STI), a single Account table might contain columns for all root and subtype attributes, with a type discriminator indicating whether a row is a generic account, creator, consumer, streamer, or subscriber. Under a class-table encoding (CTI), there may instead be separate relations such as Account, CreatorAccount, ConsumerAccount, Streamer, and Subscriber, with the leaf types reconstructed by joining upward along the hierarchy. Under a concrete-table encoding (CONCRETE), Streamer and Subscriber may each carry their inherited attributes directly, without requiring the root relation to answer leaf-specific queries.
aiDM ’26, May 31-June 05, 2026, Bengaluru, India
Nitin Kanchinadam, Aditya Menachery, Amol Deshpande
Schema Variant
Inheritance Strategy
Multi-valued Attributes
Relationship Encoding
full_sti
Single-table inheritance for both Account and Content hierarchies; subtype membership represented using discriminator columns Class-table inheritance throughout; each superclass/subclass gets its own relation with keys Concrete-table inheritance for leaf classes; inherited attributes duplicated into concrete subtype tables Class-table inheritance throughout
Prefer nested representation (arrays / JSONB) for attributes such as aliases, quality_options, and chapter_markers Fully normalized into separate child tables
Many-to-one relationships represented as foreign keys; many-to-many relationships via junction tables
full_cti
full_concrete
full_normalized
concrete_all_tables
hybrid_roots_merged
hybrid_leaves_merged mixed_sti_cti
Concrete-table inheritance for concrete subclasses, with root entities retained when needed Root and first-level hierarchy nodes merged; leaf subtypes such as Streamer, Subscriber, LiveStream, and Clip remain separate Root entities represented separately, but leaf classes merged into parent subtype tables where possible Account hierarchy uses STI, while Content hierarchy uses CTI
sti_junction_fks
STI for both major hierarchies
kitchen_sink
Mix of STI, CTI, and concrete-style encodings across the two hierarchies
Fully normalized into separate child tables Fully normalized into separate relations for every multi-valued attribute Fully normalized into separate relations Mixed: frequently queried attributes normalized; smaller collections kept nested Mixed: arrays / JSONB for compact attributes, normalized tables for larger collections Mixed normalized and nested representation Fully normalized for all multivalued attributes Mixed representation chosen attribute-by-attribute
Many-to-one relationships represented as foreign keys; many-to-many relationships via junction tables Relationship endpoints attached to concrete tables when possible; many-to-many relationships via junction tables Explicit relationship tables used aggressively, including some many-to-one relationships Explicit tables for all many-to-many relationships and selected many-to-one relationships to maximize uniformity Foreign keys for many-to-one relationships; standard junction tables for manyto-many relationships Mostly foreign-key based, with junction tables for many-to-many relationships Standard foreign keys for many-to-one relationships and junction tables for manyto-many relationships Explicit foreign keys and junction tables, minimizing nested columns despite STI Mixed encoding, including both foreignkey style and explicit tables
Table 1: Representative schema variants generated from the same social-media E/R model. All ten schemas are semantically equivalent: they represent the same entities, relationships, and attribute values, but differ in how those concepts are shredded into relational structures.
A second major source of variation is the handling of multivalued attributes, which may be normalized into separate tables (with associated performance penalty due to additional joins needed), or may be stored in nested columns such as PostgreSQL arrays or JSONB values. For example, Tag.aliases may be represented by a relation TagAlias(tag_id, alias), or it may appear as an array-valued attribute inside Tag. Likewise, Content.quality_options may become a relation ContentQualityOption(content_id, quality_option) or remain inside the Content table as a nested value. Both choices are semantically equivalent, but they lead to very different SQL. A query asking for clips that support “1080p” may compile to a join against a child table in one schema and to an array-containment predicate in another. Relationships introduce a third source of variation. For manyto-one relationships, the most common encoding is a foreign key on the many side, but this is not the only option. The producer relationship from Content to CreatorAccount, for instance, can be encoded as a producer_id attribute inside Content, or as
an explicit relation such as ProducedBy(content_id, creator_id). Similarly, the derived_from relationship between Clip and LiveStream can be represented either as a foreign key in Clip or as a standalone table. Many-to-many relationships almost always induce junction tables in a fully normalized schema, but even there there is freedom in naming, key choices, and interaction with surrounding denormalization decisions. For example, if inheritance is encoded concretely, the relationship Subscriber-subscribesto-CreatorAccount may be realized against a subtype table in one schema and against a root table with type filtering in another. In our framework, we first identify all such representational decision points by analyzing the provided E/R diagram. We then construct complete relational schemas by making different choices for those. In principle, this can produce an exponential number of schemas, and we seek a representative set of variants that are both structurally distinct and realistic. For the social-media domain, we generate ten schemas (Table 1). Some of these are “extreme” schemas that apply a single strategy consistently across the whole design. For instance, one schema
Same Data, Different Schemas: Robustness of LLM-based Text-to-SQL
may use class-table inheritance for every hierarchy and normalize every multi-valued attribute into a separate relation. Another may use single-table inheritance wherever possible and preserve multi-valued attributes using nested types. These extremes are useful because they maximize contrast and reveal whether models are sensitive to broad representational shifts. These also represent common strategies used in practice. Other schemas are hybrid, to capture the fact real databases rarely adhere to a single design principle everywhere. A designer might choose class-table inheritance for the Account hierarchy but single-table inheritance for Content; or might normalize watch histories and subscriptions, while keeping quality options or aliases in array-valued columns. Accordingly, we also include mixed schemas such as one in which the roots of both hierarchies are merged but leaf classes remain separate, another in which leaf classes are merged while the roots stay normalized, and another in which inheritance is handled by STI but many-to-many relationships and multi-valued attributes are represented by explicit junction tables.
2.3
Data and Query Generation
The conceptual instance generator operates directly over the E/R model. It first creates entity objects, assigns subtype memberships, instantiates multi-valued attributes, and then populates relationships. In the social-media domain, this means generating accounts, creators, consumers, streamers, subscribers, content objects, channels, and tags; assigning values such as verification status, average viewers, subscription tiers, or chapter markers; and then linking them through relationships such as production, hosting, publication, viewing, subscription, and sharing. The generator is parameterized by scale and skew. We can control, for example, the total number of accounts, the fraction of accounts that are creators, the fraction of creators that are streamers, the number of channels owned per creator, the average number of tags attached to each content item, or the length distributions for arrays such as watch histories or quality options. These parameters allow us to create multiple benchmark sizes without altering the logical structure of the domain. Once the conceptual instance has been created, each schema variant is populated by shredding the same objects and relationships into the target relations. The identities of conceptual objects are preserved across variants, so that a particular clip, tag, or subscriber corresponds to the same underlying entity in every database. What changes is only the representation: a fact that appears as a nullable field in one schema may be split across multiple tables in another, or may be nested inside an array or JSON structure in a third. This design gives us a clean controlled setting: the data is fixed, the question is fixed, and only the schema changes. The set of natural language questions for testing purposes can either be generated manually (by a human), or could be autogenerated using an LLM. Since users usually reason about the entities and relationships in the data when formulating their questions, in some ways, the questions are posed against the E/R model rather than any specific schema, and as a result, the same natural language question can be meaningfully posed to any of the schema variants. For the social-media domain, some examples include: asking for streamers that hosted popular live streams, channels that publish
aiDM ’26, May 31-June 05, 2026, Bengaluru, India
content from verified creators, clips derived from streams hosted on a certain platform, or subscribers who follow creators owning multiple channels. We use a relatively simple set of questions for our initial evaluation, since we could already see significant consistency issues across the schema variants; however, more complex questions can be easily added. For the social media dataset, the questions used for evaluation were: (1) Who are the 50 creators that own the largest number of channels? (2) For each creator, how many total pieces of content have they produced? (3) Who are the 50 streamers that have hosted the most live streams? (4) How many clips were derived from each live stream? (5) Who are the top 50 creators that have the largest number of subscribers? (6) For each subscriber, how many different creators are they subscribed to? (7) Which 50 channels have published the most content items? (8) How many clips has each account shared? (9) What are the 50 tags that are used on the largest number of content items? (10) For each creator, how many times has content they produced been viewed by consumer accounts (11) Who are the top 50 creators that produce content that is published on the greatest number of distinct channels? (12) How many live streams hosted by each streamer have at least one derived clip? (13) Which subscribers follow creators who own more than one channel? (14) For each channel, how many live streams and how many clips have been published on it? (15) Who are the top 50 creators that have produced content tagged with 3 or more tags? These questions were automatically generated using an LLM, with minor edits to make them less ambiguous. See Appendix A for the questions used for the other dataset.
2.4
Why These Variations Matter for LLMs
The social-media example makes clear why these schema variants can be difficult for current models to handle. A question such as “Which verified creators own channels that publish clips tagged with aliases of ‘gaming’?” is conceptually simple, but the data needed to answer it may be arranged across different sets of tables and connected in different ways. In one schema, verified may be a nullable field in a large Account table, Clip may have its own table, and tag aliases may require an unnest or JSON operator. In another schema, the same query may require joining Account, CreatorAccount, RecordedContent, Clip, ContentTag, and TagAlias. In yet another, some of these joins may disappear while others are replaced by discriminator predicates. These differences matter because text-to-SQL models often rely heavily on surface correspondences between question words and schema tokens. Equivalent schemas disturb those correspondences without changing the intended meaning. Inheritance introduces
aiDM ’26, May 31-June 05, 2026, Bengaluru, India
ambiguity about whether a concept such as subscriber or streamer should map to a table, a subtype predicate, or a join path. Multivalued attributes change not only where the relevant information is found but also what SQL operators are needed to access it. Relationship reification changes whether a connection is represented as a foreign key or as an explicit table. Our benchmark is designed precisely to test whether models can reason through such representational changes or whether their behavior is brittle.
2.5
Using the Framework for Evaluation
Suppose an E/R model yields schema variants 𝑆 1, . . . , 𝑆𝑘 , and let 𝑞 1, . . . , 𝑞𝑚 be the natural-language questions. We evaluate each text-to-SQL model on every pair (𝑞𝑖 , 𝑆 𝑗 ) using the respective APIs (the specific models used are listed in the evaluation section). For each schema-question combination, we generated the following prompt, replacing schema with the full contents of a PostgreSQL dump file containing CREATE TABLE statements for the schema and question with the natural language question: You are the world’s best natural language to SQL translator. You will be given two things: a schema for a PostgreSQL database and a natural language question. Given these two inputs, you should respond ONLY with a SQL statement that when run against the provided schema, returns the results that would answer the provided question. Again, you should ONLY answer with the SQL statement. Do not include any extra text before or after the SQL statement. When providing your query, you should provide the minimum number of columns to uniquely identify answers. For example, only return the ID of records that match your query and not other columns that are not necessary for unique identification unless the query explicitly specifies a list of columns. Return the minimal data necessary for both simple queries and aggregations. The database schema is: schema. The question is: question. As we discuss later, the list of attributes that were selected (i.e., the select clause) varied widely in the generated SQL statements. To focus on the inconsistencies arising out of schema issues, we force the models to only return IDs. The rate of mismatch was too high otherwise. Additionally, without the specification to only retrieve the ID fields, the LLMs sometimes omitted it altogether, making it hard to check for correctness or compare answers. For each generated SQL query, we record the query, whether it executes successfully, and the resulting answer. Because the same conceptual instance underlies all schemas, these answers are directly comparable across variants. This structure enables several forms of analysis. We can measure standard execution accuracy within each schema, but more importantly we can check whether the model’s behavior is stable across equivalent schemas. For example, we can compare every pair of schemas and compute how often the model returns equivalent answers for the same question, or how often it fails on one schema
Nitin Kanchinadam, Aditya Menachery, Amol Deshpande
after succeeding on another. In the experimental section, we summarize these cross-schema patterns using pairwise heatmaps, which make it easy to see which schema design choices induce the largest drops in consistency.
2.6
Providing E/R-Diagram Context
A natural question is whether the performance of the text-to-SQL systems would be improved if the original E/R diagram is provided as part of the context (we note that this is rarely available in practice). For one of the datasets (retail), we ran a second set of experiments by modifying the prompt: The Entity-Relationship diagram is: ER-Diagram Context. The schema is: schema. The associated context for the schema is: schema description. The question is: question. In addition to providing the list of entities (and their attributes), inheritance hierarchy, and relationships, we also provide a text description of the choices made in creating the relational schema (i.e., whether we used single-table inheritance, etc).
3
Preliminary Evaluation
In this section, we report the results from our experimental evaluation across four leading large language models (including Gemini, ChatGPT, Claude, and DeepSeek) across two datasets: social media and retail (Appendix A). The specific models used are noted later.
3.1
Evaluation Process
For each natural language question, we generate SQL queries from each schema and execute them in PostgreSQL. To ensure consistent comparisons across schemas, we normalize the results as follows: • Rows are stored as a list of tuples • Rows are sorted to remove ordering differences • All values are converted to strings for uniform comparison This normalization ensures that comparisons focus only on the actual result data returned, not formatting or ordering differences. We note that all of the above are important challenges for textto-SQL systems in practice. Combined with the fact that we only test against a single instance of the data, the results we discuss below constitute a best-case scenario for the text-to-SQL systems.
3.2
Pairwise Comparison Metrics
3.2.1 LLM Overview Metric. For each pair of schemas (𝑖, 𝑗) and large language model 𝑚, we compute: (Í 𝑁 1 𝑅𝑖,𝑞,𝑚 = 𝑅 𝑗,𝑞,𝑚 , if 𝑖 ≠ 𝑗 PSS(𝑖, 𝑗, 𝑚) = Í𝑞=1 𝑁 1 𝑅 = 𝑅 − 𝐸 , if 𝑖=𝑗 𝑖,𝑞,𝑚 𝑗,𝑞,𝑚 𝑖,𝑚 𝑞=1 where: • 𝑅𝑖,𝑞,𝑚 = result of schema 𝑖 on question 𝑞 using the SQL query generated by model 𝑚 • 𝑁 = total number of questions (15) • 𝐸𝑖,𝑚 is the number of queries for schema 𝑖 and model 𝑚 that resulted in invalid SQL or execution errors.
Same Data, Different Schemas: Robustness of LLM-based Text-to-SQL
aiDM ’26, May 31-June 05, 2026, Bengaluru, India
Each pairwise schema score represents the number of matched queries between each pair of two schemas. We compute this score for every pair of schemas, resulting in a matrix where each row and column correspond to each schema and each entry represents their agreement out of 15. 3.2.2 Pairwise Comparison by Question. Additionally, for each question 𝑞 and large language model 𝑚, we compute the following scores: • QuestionAgreementScore - the number of schema pairs for which an LLM returned valid SQL queries for both schemas and the queries resulted in equivalent result sets 𝑄𝐴𝑆 (𝑞, 𝑚) =
𝑁 ∑︁
1 𝑅𝑖,𝑞,𝑚 ≠ ∅ ∧ 𝑅 𝑗,𝑞,𝑚 ≠ ∅ ∧ 𝑅𝑖,𝑞,𝑚 = 𝑅 𝑗,𝑞,𝑚
(a) Gemini 3 Flash
(b) ChatGPT 5.2
(c) DeepSeek V3.2
(d) Claude Sonnet 4.6
𝑖,𝑗=1,𝑖< 𝑗
• QuestionDisagreementScore - the number of schema pairs for which an LLM returned valid SQL queries for both schemas, but the queries resulted in inequivalent result sets 𝑄𝐷𝑆 (𝑞, 𝑚) =
𝑁 ∑︁
1 𝑅𝑖,𝑞,𝑚 ≠ ∅ ∧ 𝑅 𝑗,𝑞,𝑚 ≠ ∅ ∧ 𝑅𝑖,𝑞,𝑚 ≠ 𝑅 𝑗,𝑞,𝑚
𝑖,𝑗=1,𝑖< 𝑗
• QuestionMissingScore - the number of schema pairs for which an LLM returned an invalid SQL query for either one or both of the schemas 𝑁 ∑︁ 𝑄𝑀𝑆 (𝑞, 𝑚) = 1 𝑅𝑖,𝑞,𝑚 = ∅ ∨ 𝑅 𝑗,𝑞,𝑚 = ∅) 𝑖,𝑗=1,𝑖< 𝑗
We note that: 𝑄𝐴𝑆 (𝑞, 𝑚) +𝑄𝐷𝑆 (𝑞, 𝑚) +𝑄𝑀𝑆 (𝑞, 𝑚) = 𝑁2 where 𝑁 is the number of schemas.
3.3
Overall Statistics
We visualize the pairwise agreement between the schemas using heatmaps, where the colors represent values from 0 (purple) to 15 (yellow). As shown in Figures 3 and 4, Gemini achieves the highest overall agreement across schemas, followed by Claude and ChatGPT, while DeepSeek shows the lowest agreement. Note that the diagonal line remains consistently bright in all heatmaps, confirming that each schema agrees with itself (as it should). The only instance of imperfect agreement there occurs when a query throws an error when run in PostgreSQL. In both the media and retail datasets, agreement varies significantly depending on the pair of schemas being compared. Certain schema pairs consistently produce high agreement, indicating that they are easier for models to understand. In contrast, other schema pairs exhibit substantial disagreement, reflecting differences in how relationships and entities are represented. Media: In the media dataset (Figure 3), Gemini 3 exhibits large regions of high agreement (bright yellow and green), especially among schemas like full_cti, and full_normalized. This indicates that Gemini produces highly consistent queries across structurally similar schema. However, schemas such as hybrid_leaves_merged and kitchen_sink have lower agreement rates on average. ChatGPT 5 still has clusters of high agreement, but the heatmap is more fragmented with patches of medium and low agreement. The rows and columns corresponding to full_sti and concrete_all_tables exhibit consistently darker values across many schema
Figure 3: Pairwise agreement heatmaps for all models for the media dataset. Lighter values indicate higher agreement. Excluding the diagonal entries, the average agreements were 81.6% (Gemini), 43.62% (GPT), 33.85% (DeepSeek) and 67.8% (Claude). pairings, indicating that these schema have the most disagreement. In contrast, several hybrid schema, such as hybrid_leaves_merged and hybrid_roots_merged, show relatively higher agreement. DeepSeek’s heatmap is darker overall, with fewer bright regions and many schema pairs showing low consistency. The diagonal remains bright, as expected, but the off-diagonal values are consistently lower, reflecting lower agreement across schema. Of the four models, DeepSeek tended to have the lowest agreement and the most syntactically incorrect SQL queries. Claude Sonnet 4.6 displays a more structured pattern than DeepSeek but is less uniform than Gemini. Its heatmap shows clear bands of agreement, with strong consistency among mid-range schema such as full_cti, full_normalized, and mixed_sti_cti. However, the lowest agreement is concentrated in the corners of the heatmap, particularly when comparing structurally extreme schema such as concrete_all_tables and sti_junction_fks. Retail: The retail dataset (Figure 4) shows overall higher agreements. This is attributable in part to a less ambiguous set of questions (see Appendix A) as well as use of newer models. Even then, we see significant disagrement rates, especially with GPT-5.4 and DeepSeek V4 Pro.
3.4
Agreement by Question (Media Dataset)
Next, we examine the models’ performance on each NL question using the 𝑄𝐴𝑆, 𝑄𝐷𝑆, and 𝑄𝑀𝑆 scores described earlier on. Since each LLM generated 10 queries for each NL question, one query
aiDM ’26, May 31-June 05, 2026, Bengaluru, India
(a) Gemini 3.1 Pro Preview
Nitin Kanchinadam, Aditya Menachery, Amol Deshpande
(b) ChatGPT 5.4
Figure 5: Agreement by question across schemas for each model on the media dataset.
(c) DeepSeek V4 Pro
(d) Claude Opus 4.7
aggregation questions but struggle with questions that require a deeper relational understanding of the schema. Interestingly, while Gemini showed the highest level of agreement in its SQL queries, Claude was the LLM that most consistently returned valid SQL with no invalid generated queries. On the other hand, Gemini and DeepSeek both produced a small portion of invalid statements and ChatGPT generated the most invalid SQL statements of the 4 models.
Figure 4: Pairwise agreement heatmaps for all models for the retail dataset. Lighter values indicate higher agreement. Excluding the diagonal entries, the average agreements were 88.7% (Gemini), 69.33% (GPT), 47.4% (DeepSeek) and 82.2% (Claude). The higher agreement is likely because of a less ambiguous set of questions as well as use of newer models.
We observe several mismatches that are directly caused by differences in schema design.
for each schema, there are 10 2 = 45 possible pairs of queries by the same LLM on the same NL question. The agreement by question analysis reveals that LLM consistency varies significantly based on the complexity of the query. Questions such as Q1 and Q4–Q9, which involve relatively straightforward aggregations or counting over a single relationship (e.g., counting channels, subscribers, or tags), exhibit higher agreement than the other questions for all models while for Gemini, they exhibit near perfect agreement across the 45 pairings. These queries usually require simple GROUP BY operations or minimal joins, making them less sensitive to how the schema is structured. DeepSeek, while its results exhibited mostly disagreement, had perfect agreement on Q6, which was a simple aggregation question over the subscriber-creator relationship, further showing that simple questions exhibit higher agreement. On the other hand, questions such as Q3, Q12, Q14, and Q15 show significantly higher disagreement. These queries require more complex reasoning over multiple relationships, such as linking live streams to derived clips (Q12), combining multiple entity types such as streams and clips (Q14), or enforcing complex conditions across joins such as “at least one derived clip” or “tagged with 3 or more” (Q12, Q15). As a result, models often fail to correctly reconstruct the necessary join paths or apply conditions at the appropriate stage of the query. Overall, these results demonstrate that LLMs perform reliably on simpler
3.5.1 Representing Relationships in a Separate Table. An example of mismatch caused by the schema differences occurs in Claude’s responses to Q4, which asks for the number of clips derived from each live stream. In the concrete_all_tables schema, the query operates over a table (clips_derived_from_ref) that only contains entries for live streams with at least one associated clip. Because of this, the aggregation over this table inherently excludes all live streams that have zero clips, since they do not appear in the table at all. In contrast, the full_cti schema requires joining the live_streams table with the clips table using a LEFT JOIN, which preserves all live streams and assigns a count of zero where no matching clips exist. This difference in schema design leads to different result sets: one includes only live streams that have had clips derived from them, while the other represents the full domain of live streams. The LLM fails to account for this distinction, generating queries that are locally valid for each schema but semantically inconsistent. This highlights a key limitation in current text-to-SQL capabilities, where models do not fully adapt their query logic to account for how absence of relationships is encoded in different schema designs. A similar issue is found in Gemini’s responses to Q3 for the full_concrete and hybrid_leaves_merged schemas. Q3 asks for the IDs of the streamers that have hosted the most live streams. Since the full_concrete schema has a separate live_streams table, the list of returned host_ids is accurate. On the other hand, for the hybrid_leaves_merged schema, the live_content table holds all the LiveStream records. Since LiveContent records have
3.5
Schema-Based Mismatches
Same Data, Different Schemas: Robustness of LLM-based Text-to-SQL
host_id set to NULL, the result for the hybrid_leaves_merged query says that the ID of the creator that has hosted the most live streams is NULL with 2000 hosted live streams, an obvious error. 3.5.2 Type Discrimination in Multi-Entity Tables. Another example of mismatch appears in Claude’s responses to Q4 when comparing the concrete_all_tables and hybrid_leaves_merged schemas. In the hybrid_leaves_merged schema, all live content related entities are stored in a single table (live_content), with a type attribute identifying between general live content and specifically live streams. The generated query aggregates over the entire table without filtering specifically for live streams, resulting in a significantly larger result set that includes records corresponding to non-stream and stream objects. On the other hand, the concrete_all_tables schema isolates live streams into a dedicated structure, ensuring that only valid live stream records are counted. This leads to approximately 2000 additional records in the hybrid_leaves_merged result, because the LLM fails to correctly interpret the type constraint required in schema that merge multiple entity types into a single table. This indicates a broader challenge with hybrid schema, where semantic distinctions are encoded as attribute values rather than through structural separation, requiring more precise filtering logic that LLMs often do not include. 3.5.3 Confusion from Schema Hierarchy Changes. Another schema caused mismatch occurs in Q10 when comparing the concrete_all_tables and kitchen_sink schemas. In concrete_all_tables, creators are represented through a dedicated creator_accounts table. In contrast, the kitchen_sink schema collapses parts of the account hierarchy, removing a separate creator_accounts table and instead combining all attributes into the broader accounts table while keeping streamers as a subtype. The generated query incorrectly joins through streamers instead of filtering the accounts table, which restricts the result to only streamers and not creators who aren’t streamers. As a result, the query returns only 1480 rows instead of 2843. This mismatch shows that when schema hierarchies are restructured, LLMs may confuse a subtype with the full entity set.
3.6
Non-Schema-Based Mismatches
We also observe errors that are not directly caused by schema differences. 3.6.1 Data Based Errors. A common class of errors across models consists of queries that return only zeros, empty result sets, or NULL values despite executing successfully. These failures are often due not to schema misinterpretation, but to limited awareness of the underlying data values and relationships. For example, since data values are not included in the prompt context, a model may generate filters that do not match the actual data distribution, such as using "clip" instead of the correct "Clip". Similarly, improper join conditions or aggregation logic can produce NULL values when table relationships are handled incorrectly. Such errors show that even with a correctly interpreted schema, insufficient data awareness can still lead to invalid or misleading outputs. 3.6.2 SQL Syntax Errors. A non-schema-related failure appears in Q3 for the kitchen_sink schema, where the generated query
aiDM ’26, May 31-June 05, 2026, Bengaluru, India
uses the invalid PostgreSQL expression ORDER BY COUNT() with no argument. The query therefore fails to execute and produces no output. This error reflects malformed SQL generation rather than any issue with the schema, illustrating that basic SQL correctness remains a separate source of failures even when the intended query structure is otherwise plausible. 3.6.3 Logic Based Errors. Another example of error is when the LLM returns two different responses for the same table structure as shown in DeepSeek’s responses to Q13, which asks which subscribers follow creators that own more than one channel. In concrete_all_tables and full_concrete schemas, the subscribers, subscribers_subscribes_to, and creator_accounts_owns tables have the same structure. However, while both schemas presented the model with the same structure, only the first query returned the correct results. The other query performed a threeway join between the tables and then only grouped by subscriber ID, which led to it returning all subscribers whose followed creators had more than one channel when combined rather than those whose followed creators included a single creator that had more than one channel. These errors highlight limitations in the models’ ability to correctly interpret query intent, independent of schema structure.
3.7
Performance by Schema
Overall, we observe that schemas with clear separation between entities tend to lead to higher agreement rates, while schemas that store multiple relationships or merge multiple entity types lead to significantly more mismatches. Schemas such as concrete_all_tables and full_normalized, which explicitly represent relationships through dedicated tables and foreign keys, are generally easier for models to reason with. These schemas make join paths explicit, reducing ambiguity in query generation. In contrast, hybrid and denormalized schemas such as hybrid_leaves_merged, hybrid_roots_merged, and kitchen_sink had lower agreement rates. These schema often merge multiple entity types into a single table or encode relationships through implicit attributes. This requires the model to infer filtering conditions, which are frequently omitted, leading to overcounting, undercounting, or inclusion of irrelevant records. A possible resolution is providing more context in the prompt to the model. We find that the most challenging schemas are those that deviate from standard normalization patterns by either over-merging or over-abstracting relationships. These designs require more complex reasoning, which current LLMs struggle with. On the other hand, schemas that align more closely with conventional relational modeling principles tend to produce query results with higher agreement.
3.8
Impact of Providing E/R Context
Figure 6 shows the effect of adding the full E/R context described in Section 2.6 for the retail dataset. The added domain and schema information substantially improves SQL-generation accuracy across all models. Nevertheless, the models still exhibit an overall disagreement rate of approximately 10%. As noted earlier, this is a best-case setting for text-to-SQL evaluation, since correctness is tested against a single, relatively uniform dataset; disagreement
aiDM ’26, May 31-June 05, 2026, Bengaluru, India
Nitin Kanchinadam, Aditya Menachery, Amol Deshpande
4
(a) Gemini 3.1 Pro Preview
(c) DeepSeek V4 Pro
(b) ChatGPT 5.4
(d) Claude Opus 4.7
Figure 6: Pairwise agreement heatmaps for all models for the retail dataset with additional ER & schema context. Lighter values indicate higher agreement.
rates are likely to be higher in practice, especially with messier data and more complex questions. The improvement suggests that many earlier errors were due not simply to missing table relationships, but to missing domainlevel information about what those relationships represented. For example, without being told that a query targets the full_sti schema, a model may not infer the higher-level structure implied by the schema design, particularly when subclass relationships are encoded using single-table inheritance as discussed earlier. Providing the E/R context helps expose this structure and leads to more accurate queries. Looking deeply at the 15 questions, we saw that Question 9 was the only question where all of the models had issues: “List the 3 Apparel items that are paired with the greatest number of distinct Accessories items. Return the apparel product name, category, available sizes array, and the count of accessories it is paired with. Order by accessories count descending; break ties by product name ascending.” The discrepancy is due to the LLMs performing slightly different aggregations across the three-way table joins. Specifically, in the ‘full_cti‘ schema, the apparel item with the most distinct accessories is “Leather Swimwear” with 5 distinct accessories. However, in the ‘concrete_all_tables‘ schema, the returned record for “Leather Swimwear” is only paired with 4 distinct accessories. Multitable aggregation queries are fairly complex, so it is to be expected that even an LLM that has been provided with a lot of context would struggle with these questions.
Conclusion
Our results show that current LLM-based text-to-SQL systems are not robust to schema variations. Across schemas that are logically equivalent and populated from the same underlying E/R instance, we observe substantial differences in both the generated SQL and the final answers. Taken together, the experimental results suggest that these models remain sensitive to representational details such as normalization, inheritance mapping, and relationship encoding, rather than reasoning reliably over the underlying conceptual structure. Our results also suggest that adding relevant E/R context can significantly improve the performance of these models. More broadly, this points to a gap in how text-to-SQL systems are currently evaluated. Benchmarks built around a single schema can mask an important notion of robustness that matters in practice, since real applications often admit multiple plausible relational designs for the same domain. Limitations: Our study has several limitations. First, the question set is still relatively small, and therefore does not capture the full diversity of query patterns that arise even in moderate-sized schemas. Expanding the benchmark to include more questions, more templates, and more complex reasoning patterns would provide a fuller picture of model behavior. Second, our equivalence checks are based primarily on single-shot execution results. While practical, they do not capture interactive usage of these tools [8]; they also do not have sufficient coverage and may classify queries as being equivalent when they are not. Third, our setup assumes perfectly equivalent schemas derived from a common E/R model; real-world schema evolution is often messier, involving partial overlap, added or removed attributes, and drifting semantics. Finally, lack of ground truth restricts the ability to derive stronger conclusions about the performance of the models; however, this could be remedied relatively easily by constructing ground truth queries against the simplest schemas, or the E/R diagram itself. Future Work: A natural next step is to scale up the benchmark along several dimensions: more schema families, more domains, and a much broader set of natural-language questions. It would also be valuable to study robustness at the level of individual schema transformations, in order to identify which representational changes most consistently affect model behavior. On the modeling side, our results suggest the need for text-to-SQL systems that reason over schema-invariant conceptual structure rather than over surface relational form alone, potentially through intermediate representations grounded in entities and relationships. More ambitiously, the framework introduced here could also be used not just for evaluation, but for training: equivalent schema variants offer a principled way to generate additional supervision and to encourage models to generalize across multiple relational realizations of the same underlying semantics.
References [1] Adithya Bhaskar, Tushar Tomar, Ashutosh Sathe, and Sunita Sarawagi. Benchmarking and improving text-to-SQL generation under ambiguity. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7053–7074, 2023. [2] BIRD-Team. LiveSQLBench: A dynamic and contamination-free benchmark for evaluating LLMs on real-world text-to-SQL tasks. https://github.com/birdbench/livesqlbench, 2025. Accessed: 2026-04-05.
Same Data, Different Schemas: Robustness of LLM-based Text-to-SQL
[3] Roger HL Chiang, Terence M Barron, and Veda C Storey. Reverse engineering of relational databases: Extraction of an EER model from a relational database. Data & knowledge engineering, 12(2):107–142, 1994. [4] Amol Deshpande. Beyond relations: A case for elevating to the entity-relationship abstraction. In Proceedings of the Conference on Innovative Data Systems Research (CIDR), 2025. [5] Zhongjun Ding, Yin Lin, Tianjing Zeng, Rong Zhu, Bolin Ding, and Jingren Zhou. AmbiSQL: interactive ambiguity detection and resolution for text-to-SQL. arXiv preprint arXiv:2508.15276, 2025. [6] Jonathan Fürst, Catherine Kosten, Farhad Nooralahzadeh, Yi Zhang, and Kurt Stockinger. Evaluating the data model robustness of text-to-SQL systems based on real user queries. arXiv preprint arXiv:2402.08349, 2024. [7] Zijin Hong, Zheng Yuan, Q Zhang, Hao Chen, Junnan Dong, Feiran Huang, and Xiao Huang. Next-generation database interfaces: A survey of LLM-based text-to-SQL. IEEE Transactions on Knowledge and Data Engineering, 2025. [8] Nan Huo, Xiaohan Xu, Jinyang Li, Per Jacobsson, Shipei Lin, Bowen Qin, Binyuan Hui, Xiaolong Li, Ge Qu, Shuzheng Si, Linheng Han, Edward Alexander, Xintong Zhu, Rui Qin, Ruihan Yu, Yiyao Jin, Feige Zhou, Weihao Zhong, Yun Chen, Hongyu Liu, Chenhao Ma, Fatma Ozcan, Yannis Papakonstantinou, and Reynold Cheng. BIRD-INTERACT: Re-imagining text-to-SQL evaluation via lens of dynamic interactions. In ICLR, 2026. [9] George Katsogiannis-Meimarakis and Georgia Koutrika. A survey on deep learning approaches for text-to-SQL. The VLDB Journal, 32(4):905–936, 2023. [10] Ryosuke Kohita. Exploring database normalization effects on SQL generation. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management, pages 5788–5796, 2025. [11] Wenqiang Lei, Weixin Wang, Zhixin Ma, Tian Gan, Wei Lu, Min-Yen Kan, and Tat-Seng Chua. Re-examining the role of schema linking in text-to-SQL. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6943–6954, 2020. [12] Haoyang Li, Jing Zhang, Hanbing Liu, Ju Fan, Xiaokang Zhang, Jun Zhu, Renjie Wei, Hongyan Pan, Cuiping Li, and Hong Chen. Codes: Towards building opensource language models for text-to-SQL. ACM SIGMOD, 2(3):1–28, 2024. [13] Jieyu Li, Lu Chen, Ruisheng Cao, Su Zhu, Hongshen Xu, Zhi Chen, Hanchong Zhang, and Kai Yu. Exploring schema generalizability of text-to-SQL. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1344–1360, 2023. [14] Jinyang Li, Binyuan Hui, GE QU, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin Chang, Fei Huang, Reynold Cheng, and Yongbin Li. Can LLM already serve as a database interface? a BIg bench for large-scale database grounded text-to-SQLs. In NeurIPS Datasets and Benchmarks Track, 2023. [15] Xinyu Liu, Shuyu Shen, Boyan Li, Peixian Ma, Runzhi Jiang, Yuyu Luo, Yuxin Zhang, Ju Fan, Guoliang Li, and Nan Tang. A survey of NL2SQL with large language models: Where are we, and where are we going. arXiv preprint arXiv:2408.05109, 2024. [16] Xinyu Liu, Shuyu Shen, Boyan Li, Nan Tang, and Yuyu Luo. NL2SQL-bugs: A benchmark for detecting semantic errors in NL2SQL translation. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pages 5662–5673, 2025. [17] Karime Maamari, Fadhil Abubaker, Daniel Jaroslawicz, and Amine Mhedhbi. The death of schema linking? text-to-SQL in the age of well-reasoned language models. arXiv preprint arXiv:2408.07702, 2024. [18] Mohammadreza Pourreza, Hailong Li, Ruoxi Sun, Yeounoh Chung, Shayan Talaei, Gaurav Tarlok Kakkar, Yu Gan, Amin Saberi, Fatma Ozcan, and Sercan O Arik. Chase-SQL: Multi-path reasoning and preference optimized candidate selection in text-to-SQL. arXiv preprint arXiv:2410.01943, 2024. [19] Mohammadreza Pourreza and Davood Rafiei. Din-SQL: Decomposed in-context learning of text-to-SQL with self-correction. Advances in neural information processing systems, 36:36339–36348, 2023. [20] Shetal Shah, S Sudarshan, Suhas Kajbaje, Sandeep Patidar, Bhanu Pratap Gupta, and Devang Vira. Generating test data for killing SQL mutants: A constraintbased approach. In 2011 IEEE 27th International Conference on Data Engineering, pages 1175–1186. IEEE, 2011. [21] Liang Shi, Zhengju Tang, Nan Zhang, Xiaotong Zhang, and Zhi Yang. A survey on employing large language models for text-to-SQL tasks. ACM Computing Surveys, 2024. [22] Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. Rat-SQL: Relation-aware schema encoding and linking for textto-SQL parsers. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 7567–7578, 2020. [23] Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, OctoberNovember 2018. [24] Tianshu Zhang, Kun Qian, Siddhartha Sahai, Yuan Tian, Shaddy Garg, Huan Sun, and Yunyao Li. Evoschema: Towards text-to-SQL robustness against schema
aiDM ’26, May 31-June 05, 2026, Bengaluru, India
evolution. arXiv preprint arXiv:2603.10697, 2026.
A
Retail Dataset
Figure 7 shows the E/R diagram for the retail dataset. The naturallanguage questions we use for this database are: (1) For each department, compute the total number of employees in that department and the average number of orders processed per employee, rounded to two decimal places. Include all departments that have at least one employee. Order by average orders processed descending; break ties by department name ascending. (2) For each department, report the number of employees in that department, the total number of distinct products recommended by those employees, and the total number of distinct orders processed by those employees. Return one row per department. Order by department name ascending. (3) For each of the 1,000 Electronics products, report the number of distinct Software products installable on it and the number of distinct Media products that can stream to it. Order by the sum of the two counts descending; break ties by product name ascending. (4) For each of the five concrete product types: Apparel, Accessories, Electronics, Media, and Software, report the total number of products of that type and the average price rounded to the nearest integer. Return one row per product type. Order by average price descending. (5) For every Accessories-Electronics pair, return the accessories product name, accessories category, accessories price, electronics product name, electronics model number, and electronics price. Order by accessories product name ascending, then by electronics product name ascending. (6) List every customer who has downloaded at least one Software product. Return the customer’s first name, last name, email, loyalty points, and the count of distinct Software products they have downloaded. Order by download count descending; break ties by last name ascending. (7) List every customer who has placed at least 3 orders and written at least 2 reviews. Return the customer’s first name, last name, email, order count, and review count. Order by order count descending, then review count descending, then last name ascending. (8) List every review that has a rating of exactly 5. Return the reviewer’s first name, last name, the name of the reviewed product, and the review body text. Order by reviewer last name ascending, then reviewer first name ascending. (9) List the 3 Apparel items that are paired with the greatest number of distinct Accessories items. Return the apparel product name, category, available sizes array, and the count of accessories it is paired with. Order by accessories count descending; break ties by product name ascending. (10) List the 5 customers with the highest loyalty point totals among those who have written at least one review. Return the customer’s first name, last name, email, loyalty points, and their total review count. Order by loyalty points descending; break ties by last name ascending.
aiDM ’26, May 31-June 05, 2026, Bengaluru, India
Nitin Kanchinadam, Aditya Menachery, Amol Deshpande
Product
CustOrder
User
product_id name price quantity tags[]
order_id placed_at
user_id name first_name last_name email phone_no[]
Review review_id rating body
PhysicalProduct
DigitalProduct
material dimensions
format description
Customer loyalty_points
Apparel
Accessories
Electronics
category avail_sizes[]
category material_type
model_no standards[]
Media duration
Employee department role hire_date
Software version platforms[]
Figure 7: E/R Diagram for the Retail Dataset (11) List the 5 customers with the highest total spend. Define total spend as the sum of the prices of all products that appear in orders the customer has placed: for each order the customer has placed, add the price of every product in that order (a product appearing in two different orders of the same customer is counted twice, once per order). Return the customer’s first name, last name, email, loyalty points, and total spend. Order by total spend descending; break ties by last name ascending. (12) List the 5 employees who have processed the highest number of distinct orders. Return the employee’s full name, department, role, and the count of orders they have processed. Order by order count descending; break ties by employee name ascending. (13) List the 5 employees whose recommended products span the greatest number of distinct product types, where product type is one of: Apparel, Accessories, Electronics, Media, Software. Return the employee’s full name, department,
role, the count of distinct product types represented in their recommendations, and their total number of recommended products. Order by distinct product type count descending; break ties by total recommended products descending, then by employee name ascending. (14) List the 5 most expensive Electronics products by price. Return the product name, model number, price, and the list of supported standards. Order by price descending; break ties by product name ascending. (15) List the 5 Software products that are installable on the greatest number of distinct Electronics items. Return the software name, version, price, and compatible electronics count. Order by compatible electronics count descending; break ties by software name ascending.