Under review as a conference paper at COLM 2026
IndicDB - Benchmarking Multilingual Text-to-SQL Capabilities in Indian Languages Aviral Dawar1 , Roshan Karanth2 , Vikram Goyal3 , Dhruv Kumar2 1 Independent Researcher 2 BITS Pilani, Pilani, India 3 IIIT Delhi, New Delhi, India
arXiv:2604.13686v1 [cs.CL] 15 Apr 2026
Abstract While Large Language Models (LLMs) have significantly advanced Textto-SQL performance, existing benchmarks predominantly focus on Western contexts and simplified schemas, leaving a critical gap in real-world, non-Western applications. We present IndicDB, a comprehensive multilingual Text-to-SQL benchmark designed to evaluate cross-lingual semantic parsing across diverse Indic language families. The foundational relational schemas for IndicDB are sourced from primary open-data platforms, specifically the National Data and Analytics Platform (NDAP, https://ndap.niti.gov.in/) and the India Data Portal (IDP, https:// indiadataportal.com/), to ensure the benchmark accurately reflects the structural complexity of real-world administrative data. IndicDB comprises 20 databases across 237 tables. To transform denormalized government data into complex relational structures, we utilize an iterative three-agent judge pattern (Architect, Auditor, and Refiner) to ensure structural rigor and high relational density (11.85 tables per database; joindepths up to six). The methodology employs a value-aware, difficultycalibrated, and join-enforced pipeline to systematically synthesize 15, 617 tasks encompassing English, Hindi, and five primary Indic languages. We subsequently evaluate the cross-lingual semantic parsing performance of state-of-the-art models, including Deepseek v3.2, MiniMax 2.7, Llama 3.3, and Qwen3, across seven linguistic variants to establish comprehensive performance baselines. Our results uncover a 9.00% global performance drop from English to Indic variants, highlighting a persistent ”Indic Gap” driven by increased schema-linking difficulty, greater structural ambiguity in mapping Indic language to SQL, and lack of external knowledge. IndicDB serves as a rigorous ”pressure test” for the crosslingual Text-to-SQL synthesis and semantic parsing capabilities of large language models within linguistically diverse environments. The code and benchmark are publicly available at: https://anonymous.4open.science/ r/multilingualText2Sql-Indic--DDCC/
1
Introduction
Text-to-SQL parsing aims to translate natural language questions into executable SQL queries, enabling non-expert users to interrogate relational databases without mastering query syntax. Driven by advances in Large Language Models (LLMs), performance on established benchmarks has improved dramatically: on Spider (Yu et al., 2018), top-model execution accuracy rose from 53.5% to 91.2% in recent years. The BIRD benchmark (Li et al., 2024) raised the bar with 12,751 examples over 95 large, noisy databases (33.4 GB), yet GPT-4o achieves 81.95% - 11 points behind human performance. More recently, Spider 2.0 (Lei et al., 2025) further expanded the scope to enterprise-grade data workflows spanning SQL, dialect diversity, and multi-turn interactions, reinforcing that real-world Text-to-SQL remains far from solved. 1
Under review as a conference paper at COLM 2026
A critical blind spot in this progress, however, is its overwhelmingly English-centric nature. Spider, BIRD, Spider 2.0, and WikiSQL all use English-only schemas drawn from Western contexts. MultiSpider (Dou et al., 2023) extends Spider to Chinese, Vietnamese, French, and Spanish, but inherits Spider’s relatively simple, normalized schemas. Existing Text-to-SQL benchmarks predominantly focus on English-centric, simplified schemas that fail to encapsulate the administrative and linguistic complexities inherent to the Global South. IndicDB addresses this limitation by offering a specialized evaluation suite that rigorously tests the cross-lingual semantic parsing and structural reasoning capabilities of large language models across the diverse scripts and relational frameworks of the Indian subcontinent. India’s public data ecosystem, hosted on platforms such as NDAP, IDP, ICRISAT, and IHDS, serves as a challenging evaluation testbed. These datasets feature deep administrative hierarchies (Country → State → District → Sub-District → Block → Village), resulting in foreign-key chains with a depth of six. IndicDB addresses thematic gaps in current benchmarks by incorporating domain-specific schemas for Household Surveys and Census Demography. Representative examples include an 18-table health surveillance database covering routine immunization and family planning, alongside agricultural datasets using seasonal columns such as KHARIF_SORGHUM_YIELD_KG_PER_HA. High-cardinality entity spaces with 569K unique identifiers impose extreme schema-linking demands, particularly for Indic language queries that lack lexical overlap with English-encoded column names. 1.1
IndicDB: Benchmark and Contributions
We present IndicDB, a large-scale multilingual Text-to-SQL benchmark grounded in real Indian administrative databases, evaluated across seven linguistic variants: English, Hinglish, Hindi, Bengali, Tamil, Telugu, and Marathi. Our contributions are: • Systematic Construction and Synthesis. We curate 20 PostgreSQL databases (237 tables, 7.69M rows) using a novel three-agent judge pattern (Architect, Auditor, Refiner) to produce complex star/snowflake schemas with join-depths up to six. This foundation supports 15,617 tasks synthesized via a value-aware, join-enforced pipeline across seven languages, all rigorously verified by native-speaker experts. • Comprehensive Multi-Model Benchmarking. We evaluate four state-of-the-art large language models - Llama-3.3-70B, Qwen3-8B, MiniMax-M2.7, and DeepSeek-V3.2 across zero-shot and DIN-SQL prompting methodologies (Llama Team, 2024; Qwen Team, 2025; MiniMax AI, 2026; DeepSeek AI, 2025; Pourreza & Rafiei, 2023). prompting paradigms. Our framework specifically tests the impact of external evidence augmentation SEED ( (Yun & Lee, 2025) on cross-lingual grounding in high-cardinality environments. • Characterization of the “Indic Gap.” We uncover a consistent ∼9.00% global performance drop from English to Indic variants with the most substantial deficit observed in Telugu, which exhibits a maximum decline of ∼11.02% , as detailed in Table 1. Through fine-grained error analysis, we categorize failure modes across schema complexities and linguistic nuances, providing actionable insights for improving multilingual Textto-SQL reasoning.
2
Related Work
Text-to-SQL benchmarks have progressed from the structural focus of Spider (Yu et al., 2018) to the massive volumes of BIRD (Li et al., 2024) and the enterprise-scale workflows of Spider 2.0 (Lei et al., 2025). IndicDB extends this evolution by transforming Indian government datasets into rigorous star and snowflake schemas via a 3-Agent Judge pipeline. Our benchmark incorporates multi-fact constellations and deep administrative hierarchies across six Indic languages. This framework addresses significant structural and linguistic challenges unique to multilingual semantic parsing in the Indian context. Multilingual and Cultural Grounding. To evaluate the cross-lingual capabilities of Large Language Models (LLMs), MultiSpider (Dou et al., 2023) extended foundational Text-toSQL tasks to seven languages. This was further evolved in MultiSpider 2.0 (Pham et al., 2
Under review as a conference paper at COLM 2026
Language
Avg. EX
Drop
English Hinglish Bengali Hindi Marathi Tamil Telugu
64.69% 57.82% 56.15% 55.61% 55.06% 55.81% 53.67%
– 6.87% 8.54% 9.08% 9.63% 8.88% 11.02%
Figure 1: Cross-lingual EX on IndicDB. Telugu exhibits the most significant accuracy reduction relative to English.
Figure 2: Pipeline for database schema generation.
2025), which applied enterprise-scale complexity to eight languages and identified a significant performance cliff for non-Western linguistic variants. In the broader Indian context, IndicQA (Singh et al., 2024) established a high-bar for question answering across 11 major Indian languages, proving that models struggle with the morphological richness and script complexity of Indic variants. IndicDB bridges these domains by applying enterprise-grade relational density to authentic Indian context data. Automated Task Generation. Benchmark synthesis has evolved from rule-based grammars to agentic LLM pipelines. Early benchmarks used recursive synchronous context-free grammars, which guaranteed structural correctness but produced limited linguistic and logical diversity. DSQG-Syn (Duan et al., 2025) improved this by introducing difficultyaware, question-guided SQL synthesis with iterative generation. IndicDB extends this line of work specifically for multilingual Text-to-SQL under realistic relational settings: we enforce schema-grounded join validity (FK-path-only joins) and increase hard-query coverage through controlled join/aggregation/CTE patterns.
3
The IndicDB Benchmark Construction
Building IndicDB proceeds in three phases: [1] Schema Synthesis - transforming flat government CSVs into rich relational structures, [2] Task Generation - synthesizing valuegrounded Text-to-SQL tasks, and [3] Multilingual Expansion - producing faithful Indic language variants. We detail each below. 3.1
Agentic Schema Synthesis
Indian open-data sources (NDAP, India Data Portal) distribute datasets as monolithic CSVs with 50–100+ mixed-granularity columns. We convert these into complex relational structures via a 3-Agent Judge Pattern (as shown in Figure 2) - an iterative, LLM-driven feedback loop (see prompt A.5): • Architect synthesizes normalized star or snowflake schemas by decomposing highdimensional datasets into four to ten thematic entities. These tables are categorized as Fact Tables (prefixed with FACT_) or Dimension Tables (prefixed with DIM_), with a strict limitation of fifteen columns per table. Quantitative metrics are centralized within a primary Fact Table, such as Fact_Accident_Occurrences, and linked to surrounding Dimension Tables like Dim_Time_Periods and Dim_Geographic_Regions which provide temporal or geographic context. This structural separation ensures that models must navigate complex multi-hop join operations and demonstrate precise schema-linking for accurate query synthesis. • Auditor validates the proposed architecture against design constraints such as Third Normal Form (3NF) and thematic cohesion. It evaluates relational graph complexity to en3
Under review as a conference paper at COLM 2026
Table 1: IndicDB Comprehensive Framework: Qualitative Complexity Taxonomy (Left) and Full Quantitative Benchmark Statistics (Right). Feature / Constraint Relational Depth
Easy
Medium
Hard
Category
0–1 JOIN
Exactly 1 JOIN
≥ 2 JOINs
Volume
JOIN Diversity
INNER JOIN only
INNER JOIN primarily
Diverse (INNER, LEFT, RIGHT)
Filtering Logic
Simple WHERE
Moderate (e.g., Ranges)
Complex multicolumn filters
Difficulty
Aggregation None
≤1 Clause
Required (GROUP BY)
SQL Op.
Nesting
Prohibited
Prohibited
Required (CTEs/SubQ)
SQL Tokens
< 60
60–120
> 120
Language
Agg.
Metric
Count
Total Size Unique Pairs English Hindi Indic4* Hinglish Easy Medium Hard JOIN WHERE GROUP BY ORDER BY COUNT() SUM() AVG()
15,617 3,684 3,684 1,948 8,248
Pct. / Avg. – – 30.1 w 33.0 w 24.1 w
1,737 1,055 1,539 1,085 3,484 3,278 2,441 2,289 929 809 560
29.0 w 28.6% 41.8% 29.5% 94.6% 89.0% 66.3% 62.1% 25.2% 22.0% 15.2%
*Indic-4 includes: Marathi, Bengali, Tamil, and Telugu.
sure that primary-to-foreign key linkages necessitate advanced multi-hop joins involving at least three tables. • Refiner utilizes an LLM-as-a-judge paradigm to finalize the schema by adjudicating between the Architect and the Auditor. This component standardizes column headers into canonical SQL identifiers and enforces strict data typing across all fields. The module generates a configuration file maintaining a mathematically precise one-to-one mapping back to the original denormalized source data. The agentic output was compiled directly into a Data Definition Language (DDL) file, establishing foreign key relationships and surrogate keys for hierarchical administrative data (Country → State → District → Sub-District → Village). Following rigorous manual verification by a team of database experts, this DDL file was executed and the final dataset was bulk-loaded into PostgreSQL. 3.2
Task Synthesis via Enhanced DSQG-Syn
We adopt the DSQG-Syn framework (Duan et al., 2025) for its Question-First paradigm: rather than randomly sampling columns to construct SQL (which often produces intentinconsistent pairs), it first generates domain-relevant questions across nine predefined types covering all major SQL operations (Scan, Aggregate, Filter, Sort, TopSort, Join, Except, Intersect, Union), then synthesizes grounded SQL-NLQ pairs. Our enhanced pipeline operates in four stages per database: 1. Question Generation. A schema graph is constructed from FK relationships; BFS selects connected table subsets. Domain keywords are extracted via LLM, and nine question types are generated per table group. 2. Schema Linking. A MAC-SQL–inspired selector identifies the minimal relevant subschema for each question, augmented with sample values from PostgreSQL. 3. Skeleton-Guided SQL Generation. Abstract SQL templates with placeholders are generated at three difficulty tiers (Easy 30% / Medium 40% / Hard 30%) as defined in Table 1, then filled with actual schema names and real database values, eliminating “predicate hallucination” where models fabricate filter values. 4
Under review as a conference paper at COLM 2026
Language
Question
English
Which districts in Chhattisgarh achieved a rice yield exceeding 828.55 kg per hectare during the year 1970?
Hindi
? ?? ??? ???? ?? ?? ? ?? ?? ??????? ? 1970 ? ?????? ?? ????? ? ??? 828.55 ? ?? ??? ??? ? ?? ??? ??? ? ??? ????? ??? ?? ? ??
Bengali
?? ? ?? ?? ?? ? ? ? ? ?? ??? ? ????? ??? ? ? ? ? ???? ??? ???.?? ? ?? ?? ??? ???? ? ?? ??? ? ?? ? ? ? ????? ?
Marathi
? ?? ??? ????? ?????? ? ?? ?? ?? ?? ??? ?? ??????? ????? ? ?? ???.?? ? ?? ??? ????? ? ??? ?? ? ? ?? ??? ??? ? ??? ?? ?
Tamil
? ? ? ? ? ??? ?? ? ? ? ?? ? ? ? ? ? ?1970 ? ? ?? ? ? ? ?? ? ?? ? ? ? ?? ? ? ? ? ? ? ? ? ? ? ? ?
Telugu
? ? ? ? ?? ? ?1970 ? ?? ? ? ??? ?? ? ? ? ?? ? ? ?? ????? 828.55 ?? ? ? ?? ? ?? ? ? ? ? ? ? ??? ? ? ?? ?? ?? ?? ?
Hinglish
Chhattisgarh m ein kaunse districts ne saal 1970 m ein 828.55 kg per hectare se zyada rice yield haasil kiya?
? ?? ?? ?? ?
? ? ? ? ??? ? ? 828.55 ? ?? ? ? ???? ?? ? ?? ?
SELECT T1. DI ST_NAME FROM DI M_GEOGRAPHY AS T1 JOI N FACT_CEREALS_MAJOR AS T2 ON T1. GEOGRAPHY_I D = T2. GEOGRAPHY_I D JOI N DI M_YEAR AS T3 ON T2. YEAR_I D = T3. YEAR_I D WHERE T1. STATE_NAME = ' Chhat t i sgar h' AND T2. RI CE_YI ELD_KG_PER_HA > 828. 55 AND T3. YEAR = ' 1970'
Figure 3: Example of a generated multilingual task 4. NLQ Synthesis. We prioritized linguistic vagueness during translation to ensure that Natural Language Questions (NLQs) reflect authentic human discourse rather than literal SQL-to-text mappings. By obscuring explicit schema identifiers (e.g., asking ”How many private clinics are there?” instead of ”Count the hospital IDs in the dim_facilities table where the type is Private”), the pipeline requires semantic parsers to demonstrate genuine domain understanding rather than surface-level keyword alignment. FK-Constrained Join Enforcement. We constrain the SQL generator to follow only declared foreign key paths to prevent semantically invalid joins between distinct columns such as STATE_ID and STATION_ID. This enhancement involves injecting allowed relationships into the generation prompt and applying a type-safety filter to exclude numeric operations on non-numeric columns (see prompt A.3). Task Statistics. The English dataset contains 3,684 validated natural language query and SQL pairs with a calibrated difficulty distribution as shown in Table 1. Logical and syntactic integrity is maintained through a two-tier validation protocol involving PostgreSQL execution and a manual audit by three database experts. The semantic alignment between Indic queries and SQL logic was confirmed using the Fleiss’ Kappa (κ) statistic, which yielded a coefficient of 0.84. This result indicates substantial inter-annotator agreement and validates the reliability of the human-derived labels across the multilingual corpus. 3.3
Multilingual Expansion
We expand English tasks into six additional variants: Hindi, Bengali, Tamil, Telugu, Marathi, and Hinglish (HI-EN code-switching), yielding 15,617 total tasks. We adopt an English-First approach: only the NLQ is translated while the SQL remains identical, ensuring perfect logical alignment across variants. Gemini 3 Flash serves as the primary conversion engine. (see Figure 3) Hinglish receives specialized prompting for Natural Hinglish, Roman script blending Hindi grammar with English technical terms (e.g., “Agriculture department mein kitne records hain?”), testing model performance on high-usage but low-resource linguistic patterns. (see prompt A.4) Quality Assurance and Verification. We implement a multi-stage Human-in-the-Loop (HITL) verification framework to ensure cross-lingual semantic equivalence. The pipeline operates in three phases: Phase 1: Automated Semantic Quality Screening. We evaluate the linguistic fidelity of English-to-Indic translations using the Unbabel/wmt20-comet-qe-da model (Rei et al., 2022) within a reference-free quality estimation (QE) framework. This methodology is sup5
Under review as a conference paper at COLM 2026
Mistranslation Examples
Fix
Q_en: List the names of districts that produced maize but did not produce any wheat during the year 1970, sorted alphabetically. Q_ma (bad): १९७० साली ज् वारी पिकवले ले पण गहू पिकवले नसले ल्या जिल् ह्यां ची नावे वर् णक्रमानुसार सूचीबद् ध करा. Q_ma (fix): १९७० साली मका पिकवले ले पण गहू पिकवले नसले ल्या जिल् ह्यां ची नावे वर् णक्रमानुसार सूचीबद् ध करा.
Corrected the lexical mistranslation by replacing ज् वारी (sorghum) with मका (maize). Using the wrong crop alters the query semantics and can lead to incorrect filtering in SQL generation.
Q_en: List the districts in India for the 1991 census year, ordered by the number of male workers in trade and commerce in descending order, and show only the top 10 results. Q_bn (bad): ১৯৯১ সালের আদমশু মারি অনুযায়ী, বাণিজ্ য ও ব্যবসায়ে পু রুষ শ্রমিক সংখ্যার উপর ভিত্ তি করে উর্ধ্রক্রমে সাজানো ভারতের জেলাগু লির তালিকা দিন এবং শু ধুমাত্ র শীর্ষ ১০টি ফলাফল দেখান। Q_bn (fix): ১৯৯১ সালের আদমশু মারি অনুযায়ী, বাণিজ্ য ও ব্যবসায়ে পু রুষ শ্রমিক সংখ্যার উপর ভিত্ তি করে অবরোহ ক্রমে সাজানো ভারতের জেলাগু লির তালিকা দিন এবং শু ধুমাত্ র শীর্ষ ১০টি ফলাফল দেখান।
Corrected the ordering direction by replacing উর্ধ্রক্রমে (ascending order) with অবরোহ ক্রমে (descending order) to match the intended sorting in the query.
Table 2: Mistranslation examples and corresponding fixes
ported by evidence that neural-based metrics achieve a higher correlation with human judgments (r > 0.40) than traditional lexical overlap methods (Sai B et al., 2023). The quality score is predicted by a neural network f that processes the interaction between source embeddings es and hypothesis embeddings eh : COMETQE (s, h) = f (es , eh , |es − eh |, es ⊙ eh ) The translated dataset achieved a mean COMET score of µ = 0.820 with a standard deviation of σ = 0.0834 (shown in A.7). Phase 2: Statistical Thresholding and Expert Review. To ensure the logical integrity of the dataset, we implemented a baseline deviation filter to identify statistically anomalous samples. We conducted a sensitivity analysis over various thresholds and selected τ = µ − 1σ as the primary operating point for targeted review. This threshold corresponds to a score of 0.737 and flags 944 tasks, representing 7.91% of the analyzed set. All instances falling below this limit are classified as high-risk and undergo comprehensive manual review by nativespeaking linguists. This procedure concentrates expert auditing on the empirical lower tail of the score distribution to address potential semantic drift or degraded translation quality. Phase 3: Targeted Error Correction. The systematic audit of the flagged instances was conducted by a panel of three translation experts who identified two primary categories of recurrent errors, which together accounted for the majority of the reviewed samples. • Lexical Entity Divergence (approximately 31.2% of flagged instances): This error typology involved the mistranslation of domain-specific entities, such as agricultural varieties or regional administrative designations, which directly compromised the precision of SQL WHERE clause filters. • Logical Directional Inversion (approximately 29.8% of flagged instances): We observed instances where sorting directives were erroneously swapped in the target script (for example, a request for descending order being translated as ascending), necessitating a manual correction of the corresponding ORDER BY logic. Beyond these primary categories (example shown in Table 2), we also identified instances of prompt leakage, where specific English instructions or system-level directives were inadvertently retained in the final Indic translation. Following a collaborative review process among the three experts to reconcile any initial discrepancies, a final inter-annotator agreement of 91% was reached for all classifications and subsequent manual corrections. By systematically addressing these failures, we ensure that the performance disparities reported 6
Under review as a conference paper at COLM 2026
in our benchmarks reflect the reasoning limitations of the models rather than foundational translation errors.
4
Experiments
4.1
Experimental Setup
Language Selection. We evaluate the robustness of text-to-SQL systems across a linguistically diverse set of Indic and code-mixed settings. Our study encompasses seven languages: English, which serves as the baseline; five typologically diverse Indic languages - Hindi, Marathi, Bengali, Tamil, and Telugu; and Hinglish, a code-mixed Hindi–English variant that reflects real-world usage in multilingual contexts. To ensure a controlled comparison across languages, we keep the underlying database schema fixed and vary only the natural language queries via translation. Models We evaluate a diverse set of recent large language models that span a range of architectural designs and model scales. Our evaluation includes Llama 3.3 70B Instruct (70B parameters) (Llama Team, 2024), Qwen3 8B (8B parameters) (Qwen Team, 2025), decoderonly transformers; DeepSeek V3.2 (DeepSeek AI, 2025), a mixture-of-experts transformer with a total parameter count exceeding 671B (with a smaller subset activated per token); and MiniMax M2.7 (MiniMax AI, 2026), a recent large language model with agent-oriented capabilities and self-evolving training mechanisms. All models are used off-the-shelf without any task-specific fine-tuning. Prompting Strategies We evaluate model performance under two prompting strategies: Zero-shot prompting and the DIN-SQL (Pourreza & Rafiei, 2023) framework. DIN-SQL decomposes text-to-SQL generation into a sequence of structured intermediate steps, including [1]schema linking, [2]clause-wise SQL construction, and [3]iterative self-correction, which together improve reasoning and execution accuracy (see prompts A.2.1, A.2.2, A.2.3, A.2.3). We further augment DIN-SQL with evidence files to provide explicit grounding signals during generation. Zero-shot prompting is evaluated across two settings based on the inclusion of auxiliary evidence files. These files, generated for each language via the SEED (Yun & Lee, 2025) approach, provide schema linking cues, column values, and SQL generation hints. We evaluate the evidence-augmented DIN-SQL variant and perform ablation studies to determine the impact of these auxiliary signals. To ensure experimental parity, we utilize identical prompt templates and a fixed number of in-context examples for all languages. All trials employ deterministic decoding with a temperature of 0 and top-p of 1. DIN-SQL is selected for its structured decomposition, which facilitates improved schema grounding and compositional reasoning. The integration of evidence files strengthens the alignment between natural language and database structures, resulting in consistent execution accuracy gains in multilingual settings characterized by high lexical variation. Evaluation metrics We evaluate performance using Execution Accuracy (EX) (Yu et al., 2018; Li et al., 2024), which measures whether the predicted SQL query produces the same result as the ground truth when executed on the database. For each example ( j ), let Ŝ j denote the predicted query and S∗j denote the corresponding gold query. The metric is defined as: EX =
] 1 m [ 1 Exec(Ŝ j ) = Exec(S∗j ) ∑ m j =1
where Exec(·) returns the result set from executing the query on the database. 7
Under review as a conference paper at COLM 2026
Method
Model
English
Hindi
Bengali
Marathi
Tamil
Telugu
Hinglish
DIN-SQL (w/ evidence)
LLaMA 3.3 70B Instruct Qwen3 8B DeepSeek V3.2 Minimax 2.7
66.10% 55.05% 69.07% 62.86%
57.97% 52.65% 64.45% 59.05%
62.21% 51.14% 65.36% 62.73%
54.52% 49.36% 63.60% 63.67%
58.09% 49.38% 66.67% 68.07%
57.98% 49.98% 63.61% 62.00%
65.07% 51.06% 66.17% 65.10%
Zero-shot (w/o evidence)
LLaMA 3.3 70B Instruct Qwen3 8B DeepSeek V3.2 Minimax 2.7
58.06% 52.17% 69.32% 59.51%
44.24% 38.39% 57.66% 53.30%
45.14% 37.52% 58.06% 50.95%
42.20% 36.40% 56.66% 52.00%
42.30% 34.23% 56.04% 49.39%
39.46% 34.50% 52.94% 48.23%
43.13% 38.58% 60.53% 50.21%
Zero-shot (w/ evidence)
LLaMA 3.3 70B Instruct Qwen3 8B DeepSeek V3.2 Minimax 2.7
73.31% 57.97% 74.93% 76.91%
61.27% 42.44% 67.73% 67.13%
58.48% 57.53% 66.65% 67.11%
57.14% 52.97% 64.06% 63.34%
62.32% 58.10% 66.54% 64.16%
47.65% 45.46% 61.65% 57.72%
63.69% 53.98% 70.89% 70.83%
0.00% 0.00%
-13.82% -9.08%
-13.34% -8.54%
-15.86% -9.63%
-17.94% -8.88%
-18.60% -11.02%
-14.93% -6.87%
Max Performance Drop Avg. Drop per Language
Table 3: Execution accuracy (EA) across languages for different prompting strategies. The bold values indicate the highest performance for each method and model configuration, while the final rows quantify the performance degradation across the Indic linguistic spectrum. Incorrect Columns 5%
Incorrect Entities 6%
Setting
English
Hindi
Bengali
Marathi
Tamil
Telugu
Without evidence With evidence
45.00% 69.07%
39.75% 64.45%
40.61% 65.36%
36.75% 63.60%
39.75% 66.67%
37.90% 63.61%
∆ (Gain)
+24.07%
+24.70%
+24.75%
+26.85%
+26.92%
+25.71%
Missing GROUP BY 14%
Group-By 21%
Failures
Incorrect Agg. 3% Missing Aggregation 5%
Aggregation 8%
Other 13% Join 12% Logical 6%
Figure 4: Execution accuracy (EA) with and without evidence file augmentation across languages. 4.2
Condition / Filter 15%
Projection 7%
Incomplete GROUP BY 7%
Schema Linking 20%
Missing / Extra Conditions 8%
Incorrect Condition 7%
Incorrect Tables 9%
Join Tables 7%
Nested 11%
Join Keys 5%
Incorrect Subquery 5%
Missing Subquery 6%
Figure 5: Distribution of error categories on the benchmark.
Experimental Results and Analysis
4.2.1 Main Results We present the main results across languages in Table 3. Across all 15,617 tasks and seven linguistic variants, we observe a global average performance drop of 9.00% relative to English, indicating a consistent cross-lingual degradation in text-to-SQL performance. To provide a method-agnostic view, we compute the average accuracy across both prompting strategies and all models for each language. Hindi and Bengali exhibit moderate degradation −9.08% and −8.54%, respectively, while Marathi and Tamil show comparable drops of −9.63% and −8.88%. Telugu exhibits the largest drop at −11.02%, whereas Hinglish shows the smallest drop −6.87% and achieves performance closest to English. These results indicate that multilingual performance varies significantly across languages, with consistent degradation observed relative to English. To better understand the causes of multilingual performance degradation, we analyze model errors across all languages and focus on the two dominant categories: schema linking errors and aggregation/group-by errors, which together account for the majority of failures (See Figure 5). Schema linking errors (20%) originate from the misalignment of natural language mentions with database elements such as tables, columns, and entities. These errors are most pronounced in Telugu, which exhibits a performance decline of 11.02% due to linguistic distance and morphological variation from English. This divergence between query tokens 8
Under review as a conference paper at COLM 2026
and schema representations complicates semantic grounding. Conversely, Hinglish shows the smallest performance drop and fewer linking errors. The presence of English tokens within code-mixed queries facilitates direct alignment with schema elements, reducing ambiguity and improving grounding accuracy. Aggregation and group-by errors (28%) represent the largest category of structural mistakes, primarily involving missing or incomplete GROUP BY clauses and incorrect aggregation behavior. These errors reflect limitations in compositional reasoning, where models fail to correctly infer aggregation constraints from the query. This challenge is amplified in multilingual settings, where variations in how quantitative or comparative intent is expressed can obscure the underlying structure of the query. As a result, models often capture the relevant entities but fail to construct the correct SQL operations. (We have shown some case-studies in 6) 4.2.2
Ablation study - Use of evidence
Evaluation of DeepSeek V3.2 across 6,245 tasks spanning seven languages reveals that structured signals yield a consistent execution accuracy improvement of +24% to +27% (Table 4). Analysis in Figure 8 suggests that these gains arise from enhanced semantic grounding that aligns natural language queries with canonical database values. Evidence files act as a structural scaffold for SQL synthesis by improving compositional reasoning for aggregation logic and complex join conditions. Significant performance increases are observed in Marathi (+27.5%), Tamil (+27.3%), and Telugu (+25.7%), whereas English demonstrates a more modest improvement of +23.7%. This suggests that the efficacy of these files is highest when addressing substantial representational disparities between natural language and database schemas.
5
Limitations and Future Directions
This study provides an initial exploration of large language model cross-lingual capabilities in the Indian context. Future efforts will prioritize expanding linguistic coverage to include a broader array of low-resource Indic languages beyond the seven variants currently evaluated. While this investigation utilizes administrative data from the National Data and Analytics Platform, subsequent research will incorporate heterogeneous domains and unnormalized structures to evaluate model robustness. There is significant potential to utilize supervised fine-tuning and retrieval-augmented generation to address performance deficits in high-cardinality environments. Furthermore, future benchmark iterations will implement automated methods to mitigate logical inversions and lexical divergences identified during error analysis. Finally, further research is required to examine how multi-turn interactions and agentic workflows impact the reliability of multilingual Text-to-SQL synthesis across diverse relational frameworks.
6
Conclusion
We presented IndicDB, a comprehensive benchmark for evaluating cross-lingual semantic parsing within the complex administrative landscape of the Indian subcontinent By employing an iterative three-agent judge pattern, comprising Architect, Auditor, and Refiner agents, we transformed denormalized public data into mathematically rigorous star and snowflake schemas across 237 tables. The resulting 15,617 tasks were validated through a multi-stage Human-in-the-Loop framework, utilizing COMET scores and expert linguistic audits to ensure logical and semantic integrity. Our empirical analysis across state-of-the-art models uncovered a 9.00% global performance drop, characterizing a persistent Indic Gap driven by schema-linking difficulties and structural reasoning deficits. Finally, we demonstrated that external evidence augmentation effectively narrows this deficit, indicating that achieving parity in Text-to-SQL synthesis requires models to move beyond surface-level translation toward a deeper understanding of diverse relational frameworks and culturally specific domain knowledge. 9
Under review as a conference paper at COLM 2026
Acknowledgments The authors acknowledge the use of AI tools such as ChatGPT, Claude, and Gemini for improving the presentation and grammar of this paper. All the results, analysis, and proposed techniques remain a concrete representation of the author’s contributions. The authors take full responsibility for the contents in this paper.
References DeepSeek AI. Deepseek-v3.2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556, 2025. Longxu Dou, Yan Gao, Mingyang Pan, Dingzirui Wang, Wanxiang Che, Dechen Zhan, and Jian-Guang Lou. Multispider: towards benchmarking multilingual text-to-sql semantic parsing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 12745–12753, 2023. Shaoming Duan, Youxuan Wu, Chuanyi Liu, Yuhao Zhang, Zirui Wang, Peiyi Han, Shengyuan Yu, Liang Yan, and Yingwei Liang. DSQG-syn: Synthesizing high-quality data for text-to-SQL parsing by domain specific question generation. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Findings of the Association for Computational Linguistics: NAACL 2025, pp. 2971–2989, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176-195-7. doi: 10.18653/v1/2025. findings-naacl.162. URL https://aclanthology.org/2025.findings-naacl.162/. Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin Su, Zhaoqing Suo, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, et al. Spider 2.0: Evaluating language models on real-world enterprise text-to-sql workflows. In The Thirteenth International Conference on Learning Representations (ICLR), 2025. URL https://openreview.net/forum?id= XmProj9cPs. Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, et al. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sql. In Advances in Neural Information Processing Systems (NeurIPS), 2024. AI at Meta Llama Team. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. MiniMax AI. Minimax m2.7: Early echoes of self-evolution. https://www.minimax.io/news/ minimax-m27-en, 2026. Accessed: 2026. Khanh Trinh Pham, Thu Huong Nguyen, Jun Jo, Quoc Viet Hung Nguyen, and Thanh Tam Nguyen. Multilingual text-to-sql: Benchmarking the limits of language models with collaborative language agents. In Australasian Database Conference, pp. 108–123. Springer, 2025. Mohammadreza Pourreza and Davood Rafiei. Din-sql: Decomposed in-context learning of text-to-sql with self-correction. arXiv preprint arXiv:2304.11015, 2023. Qwen Team. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. T. Martins. CometKiwi: IST-unbabel 2022 submission for the quality estimation shared task. In Philipp Koehn, Loïc Barrault, Ondřej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Tom Kocmi, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Marco Turchi, and Marcos Zampieri (eds.), Proceedings of the Seventh Conference on Machine Translation (WMT), pp. 634–645, Abu Dhabi, United Arab Emirates (Hybrid), December 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.wmt-1.60. URL https://aclanthology.org/2022.wmt-1.60/. 10
Under review as a conference paper at COLM 2026
Ananya Sai B, Tanay Dixit, Vignesh Nagarajan, Anoop Kunchukuttan, Pratyush Kumar, Mitesh M. Khapra, and Raj Dabre. IndicMT eval: A dataset to meta-evaluate machine translation metrics for Indian languages. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14210–14228, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.795. URL https://aclanthology.org/2023.acl-long.795/. Harman Singh, Nitish Gupta, Shikhar Bharadwaj, Dinesh Tewari, and Partha Talukdar. Indicgenbench: A multilingual benchmark to evaluate generation capabilities of llms on indic languages. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11047–11073, 2024. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 3911– 3921, 2018. Janghyeon Yun and Sang-goo Lee. Seed: Enhancing text-to-sql performance and practical usability through automatic evidence generation. In Proceedings of the IEEE ICDE Workshops (ICDEW), 2025.
11
Under review as a conference paper at COLM 2026
A
Appendix
A.1
Case study Examples
Case Studies
Question (Hindi)
GOLD SQL
1991 ? ?? ??? ??? ?? ? ?? ??? ?? ??? ??, ?? ? ? ????? ???? ? ? ???? ? ? ?? ?? ???? ? ? ??? ?? ? ??? ??? ?? ? ??5000 ? ?? ??? ??? ? ? ? ???? ?? ?? ? ? ?? ? ?? ???? ?? ? ?? 1000 ? ?? ??? ??, ??????? ????? ?? ? ?? ? ???? ? ?? ?? ? ?? ??? ? ??? ?? ?? ?? ??? ??? ? ?? ? ?? ? (Based on the 1991 census data, list the nam es of places where the total num ber of occupied houses exceeded 5000 and the total fem ale Scheduled Tribe population exceeded 1000, and arrange the results in alphabetical order by place nam e.) SELECT (...) GROUP BY T1.location_nam e HAVING SUM(T2.num ber_of_occupied_residential_houses) > 5000 AND SUM(T3.fem ale_scheduled_tribe_population) > 1000 ORDER BY T1.location_nam e
Predicted SQL
SELECT (...) WHERE T2.num ber_of_occupied_residential_houses > 5000 AND T3.fem ale_scheduled_tribe_population > 1000 AND T2.census_year = 1991 AND T3.census_year = 1991 ORDER BY T1.location_nam e
Error Reason
Missing GROUP BY clause
Question (Telugu)
2022 ? ?? ? ? ??? ??? ?? ? ? ? ?5,000 ??? ?? ?? ??? ? ? ? ? ?? ? ? ??? ??? ? ?? ?? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ???? ? ?? ?? ? , ? ?? ? ? ?? ?? ? ? ?? ??? ? ? ? ? ? ??? ?? ? ? ?? ? . (Identify the m onths that belong to the year 2022 or recorded m ore than 5,000 allopathy outpatient visits, and order the results alphabetically.)
GOLD SQL
SELECT (...) WHERE T2.year_value = 2022 OR T3.outpatient_attendance_allopathic > 5000 ORDER BY T1.m onth_nam e
Predicted SQL
SELECT (...) WHERE T2.year_value = 2022 AND T3.outpatient_attendance_allopathic > 5000 ORDER BY T1.m onth_nam e
Error Reason
Wrong Logical Operator
Question (Bengali)
? ? ? ? ? ? ??? ?? ? ?? ? ?? ? : ???? ??? ? ??? ?? ? ? ??? ? ?? ? ? ? ? ?????? ? ? ? ? ????? ??? ? ? ?? ? ?? ? ??? ? ??? ? ?? ????? ? ??? ? ?? ?? ? ?? ? ? ? ?? ? ??? ? ? ???? ? ? ? ?? ?? ? ?? ?? ????? (List the top five sub-districts in the Andam an and Nicobar Islands with the highest total num ber of first-dose Vitam in A adm inistrations recorded in 2021.)
GOLD
SELECT (...) JOIN fact_child_im m unization_specialized AS T3 ON T1.geography_key = T3.geography_key JOIN dim _tim e AS T2 ON T3.tim e_key = T2.tim e_key (...)
Predicted SQL
SELECT (...) JOIN FACT_CHILD_IMMUNIZATION_SPECIALIZED AS T2 ON T1.GEOGRAPHY_ID = T2.GEOGRAPHY_ID JOIN DIM_TIME AS T3 ON T2.TIME_ID = T3.TIME_ID (...)
Error Reason
Incorrect JOIN key
Figure 6: Case studies illustrating lexical and structural errors across languages. Figure 6 presents multilingual case studies highlighting failure modes where models fail to map natural language queries to correct SQL structures despite accurate translations. In the Hindi instance, a Missing GROUP BY error occurs as the model employs row-level filters instead of the necessary aggregation and grouping logic. The Telugu case demonstrates a Wrong Logical Operator error, where a disjunctive requirement is incorrectly predicted as an AND condition, illustrating the difficulty of preserving logical semantics across languages. In the Bengali example, an Incorrect JOIN key error reveals a failure in schema linking, as the model identifies correct tables but fails to align their relationships accurately. These patterns indicate that performance degradation is primarily caused by failures in structural reasoning and schema alignment rather than translation errors. The linguistic diversity of multilingual queries often obscures the cues required for SQL operator mapping, leading to systematic errors in aggregation, logical reasoning, and join conditions. A.2
Prompt Template
A.2.1
Prompt for Schema Linking
Schema Linking Prompt You are a smart and responsible PostgreSQL expert. Assist in identifying the database tables and columns involved in natural language queries. ### Instruction:
12
Under review as a conference paper at COLM 2026
Your task is to analyze the provided database schema, comprehend the posed question, and leverage the hint to identify which tables are needed to generate a SQL query for answering the question. The returned JSON format must strictly adhere to the following specifications: { "tables": [ { "table": "table name", "columns": ["relevant column 1", "relevant column 2", ...] }, ... ] } Each relevant column must belong to its respective table, and the output JSON object must be wrapped in a code block using ```json```. Please note that each table and column comes with detailed description information and example values for reference. ### Database schema: {schema_str} ### User question: {query} ### Hint: {evidence}
A.2.2
Prompt for Basic SQL Generation Pipeline
Basic SQL Generation Pipeline You are a intelligent and responsible PostgreSQL expert. ### Instruction: You need to read the database schema to generate SQL query for the user question. The outputted SQL must be surrounded by ```sql``` code block. ### Database Schema: {schema} ### Hint: {evidence} ### User Question: {query} The outputted SQL must be surrounded by ```sql``` code block.
A.2.3
Prompt for Divide-and-Conquer Chain-of-Thought
Divide Prompt You are a smart and responsible PostgreSQL expert. Given a database schema and a question, users want to know the corresponding SQL query. Your task is to understand the database schema and question, and decompose the question into sub-
13
Under review as a conference paper at COLM 2026
questions so user can better understand it. Each sub-question is enclosed in «». Here is an example for reference: ### Example: ## Given the database schema: {example_database_schema} ## Question: {example_question} ## Decompose the Question into sub-questions, each sub-question is enclosed in «»: Sub-question 1: «{sub question 1}» Sub-question 2: «{sub question 2}» Sub-question 3: «{sub question 3}» ### Your task: decompose the question into sub-questions. ## Given the database schema: {schema} ## Question: {query} ## Hint: {evidence} ## Decompose the Question into sub-questions, each sub-question is enclosed in «»: Conquer Prompt You are a smart and responsible PostgreSQL expert. Given a database schema and a question, your tasks are: 1. Parse user questions: Use natural language processing (NLP) techniques to parse user questions and extract query requirements and conditions. 2. Analyze database schema: Based on the database schema, understand the fields and relationships of the table, and build the basic framework of the SQL query. 3. Check sample data: Analyze the data characteristics based on the first three rows of the table values to help determine how to construct query conditions and filter results. 4. Generate SQL query: Based on user questions, query requirements and conditions, database schema, and sample data, build a complete SQL query. 5. Verification and optimization: Check whether the generated SQL query is logical and optimize it if necessary. ### Database Schema: {schema} ### Examples: {examples} ### Question: {query}
14
Under review as a conference paper at COLM 2026
### Hint: {evidence} Please generate the corresponding SQL query. ```sql``` code block.
SQL must be surrounded by
Assemble Prompt You are a smart and responsible PostgreSQL expert. Given a database schema and a question, users want to know the corresponding SQL query. ### Instructions: We have decomposed the main question into sub-questions, now your task is based on the SQL querys for corresponding sub-questions, assemble the final SQL for the main question: 1. Understand the database schema and the main question; 2. Read and analyze each sub-question and corresponding SQL query; 3. Analyze the relationship between sub-questions and the main question in order to assemble them properly; 4. Generate the final SQL for the main question and optimize it if needed. ### Database Schema: {schema} ### Main question: {query} ### Hint: {evidence} ### Sub-questions and corresponding output, including SQL querys and explanation: {subs} Based on the SQL querys for corresponding sub-questions, generate the final SQL for the main question in the end of your response, SQL must be surrounded by ```sql``` code block. A.2.4
DIN-SQL Prompt
Debugger Prompt For the given question, use the provided tables, columns, foreign keys, and primary keys to fix the given PostgreSQL QUERY for any issues. If there are any problems, fix them. If there are no issues, return the PostgreSQL QUERY as is. Use the following instructions for fixing the SQL QUERY: 1) Pay attention to the columns that are used for the JOIN by using the Foreign_keys. 2) Use DESC and DISTINCT when needed 3) Pay attention to the columns that are used for the GROUP BY statement 4) Pay attention to the columns that are used for the SELECT statement.
15
Under review as a conference paper at COLM 2026
5) Only change the GROUP BY clause when necessary (Avoid redundant columns in GROUP BY). 6) The question may be in non-english language, the sql query has to be in english. 7) Don’t include back-ticks around table names or columns names in the SQL query
A.3
DSQG-Syn enhanced prompts DSQG-Syn Skeleton Generation Prompt Please generate {num_skeletons} SQL templates based on the given question and schema. Ensure that a mix of SQL clauses are included, such as SELECT, FROM, JOIN, WHERE, GROUP BY, ORDER BY, and HAVING.
### Instruction: 1. Use ‘col_#‘ for column names. 2. Use ‘table_#‘ for table names. 3. Use ‘value_#‘ for constant values. 4. Follow the difficulty guidance in {difficulty_instructions}. ### Prompt Body: P l e a s e g e n e r a t e { num_skeletons } SQL t e m p l a t e s based on t h e given q u e s t i o n and schema . Ensure t h a t a mix o f SQL c l a u s e s a r e included , such as SELECT , FROM, JOIN , WHERE, GROUP BY , ORDER BY , and HAVING. Use p l a c e h o l d e r s f o r s p e c i f i c t a b l e and column names as f o l l o w s : 1 . Use c o l _ # f o r column names . 2 . Use t a b l e _ # f o r t a b l e names . 3 . Use value_ # f o r c o n s t a n t v a l u e s . { difficulty_instructions } Example : Input : { " q u e s t i o n " : "Show me t h e r e d s h i f t o f s p e c t r o s c o p i c o b j e c t with s u b c l a s s o f STARFORMING" } Schema : CREATE TABLE s p e c o b j ( s p e c o b j i d number Example Values [ ( Decimal ( ' 2 9 9 4 8 9 9 5 2 3 2 2 8 4 0 5 7 6 ' ) , ), ...] , s u b c l a s s t e x t Example Values [ ( None , ) , ( 'BROADLINE' , ) , ( 'STARFORMING' , ) ] , z number Example Values [ ( 7 . 0 1 1 2 4 , ) , (0.00415325 ,) , (0.00415325 ,)] , . . . . . . primary key ( s p e c o b j i d ) ) Output :
16
Under review as a conference paper at COLM 2026
{ " templates " : [ { " t e m p l a t e " : " SELECT c o l _ 0 FROM t a b l e _ 1 WHERE c o l _ 0 = value_0 " } , { " t e m p l a t e " : " SELECT c o l _ 0 FROM t a b l e _ 1 WHERE c o l _ 1 > value_0 " } , ... ] } The " t e m p l a t e s " l i s t must c o n t a i n e x a c t l y { num_skeletons } it em s . Now, apply t h e same t r a n s f o r m a t i o n t o t h e q u e s t i o n below . Do not l e t s p e c i f i c t a b l e names , column names , or c o n s t a n t v a l u e s ( l i k e " d e s c r i p t i o n " , " name " , "GALAXY" , or "BROADLINE " ) appear i n t h e t e m p l a t e . Input : { " question " : " { question . question_text } " } Schema : { schema_str } Output i n JSON format : { " templates " : [ { " template " : " . . . " } , ... ] } The " t e m p l a t e s " l i s t must c o n t a i n e x a c t l y { num_skeletons } it em s .
DSQG-Syn SQL Generation Prompt You are an expert in a specific domain and a PostgreSQL SQL expert. ### Instruction: You are provided with: 1. An SQL query template. 2. A question that the query needs to answer. 3. The schema of the relevant database. 4. Optional sample values from the database columns. You must: 1. Use only the provided schema. 2. Use only foreign-key-valid join predicates. 3. Use only provided sample values for literal filters. 4. Respect type safety for numeric and non-numeric columns. 5. Output JSON only. ### Prompt Body: You a r e an e x p e r t i n a s p e c i f i c domain and a PostgreSQL SQL expert . You a r e provided with : 1 . An SQL query t e m p l a t e . 2 . A q u e s t i o n t h a t t h e query needs t o answer . 3 . The schema o f t h e r e l e v a n t d a t a b a s e . { s a m p l e _ v a l u e s _ s t r } Your t a s k i s t o : 1 . S t r i c t l y use t h e i n f o r m a t i o n from t h e provided schema t o complete PostgreSQL q u e r i e s . Ensure t h a t a l l n e c e s s a r y
17
Under review as a conference paper at COLM 2026
t a b l e names , column names , and c l a u s e s ( such as FROM and JOIN ) come from t h e schema only . 2 . ∗∗CRITICAL JOIN RULE∗ ∗ : Use ONLY f o r e i g n −key− v a l i d j o i n p r e d i c a t e s . A JOIN c o n d i t i o n must e x a c t l y match one o f t h e allowed FK r e l a t i o n s h i p s l i s t e d below ( d i r e c t i o n can be r e v e r s e d ) . Do NOT j o i n s e m a n t i c a l l y u n r e l a t e d IDs ( e . g . , STATE_ID = STATION_ID ) j u s t because data t y p e s match . 3 . ∗∗CRITICAL ∗ ∗ : When using l i t e r a l v a l u e s i n WHERE, HAVING, IN , or o t h e r f i l t e r c l a u s e s , you MUST use ONLY t h e sample v a l u e s provided above . Do NOT make up or h a l l u c i n a t e v a l u e s . This e n s u r e s t h e gener ate d q u e r i e s w i l l r e t u r n a c t u a l r e s u l t s when executed a g a i n s t t h e d a t a b a s e . 4 . ∗∗CRITICAL TYPE RULE∗ ∗ : Use numeric o p e r a t o r s / a g g r e g a t e s only on numeric columns . − AVG/SUM r e q u i r e numeric columns . − Numeric comparisons ( > , >= , < , <=) r e q u i r e numeric/date columns . − For t e x t columns , use e q u a l i t y / i n e q u a l i t y , IN , LIKE , I S NULL, COUNT, GROUP BY . − Do not c a s t t e x t columns t o numeric u n l e s s v a l u e s a r e guaranteed numeric i n schema c o n t e x t . 5 . Avoid i n t r o d u c i n g any t a b l e names , column names , or o t h e r elements t h a t a r e not e x p l i c i t l y d e f i n e d i n t h e schema . 6 . Generate { num_sqls } PostgreSQL SQL q u e r i e s t h a t a r e d i r e c t l y r e l a t e d t o t h e given q u e s t i o n and f i t t h e SQL query t e m p l a t e . 7 . Use PostgreSQL −compatible s ynt ax only . 8 . Keep t h e output i n JSON format . { difficulty_instructions } Allowed FK r e l a t i o n s h i p s f o r JOINs : { fk_constraints_str } Numeric columns ( s a f e f o r AVG/SUM and numeric comparisons ) : { numeric_cols_str } Non−numeric columns ( do NOT use AVG/SUM or numeric comparisons ) : { non_numeric_cols_str } Example : Input : SQL Query Template : SELECT c o l _ 1 , c o l _ 2 FROM t a b l e _ 1 JOIN t a b l e _ 0 WHERE c o l _ 3 = value_0 ; Question : What a r e t h e names and d e s c r i p t i o n s o f t h e d i f f e r e n t t y p e s o f photos a s s o c i a t e d with o b j e c t s i n t h e a s t r o p h y s i c a l classifications from t h e s p e c o b j t a b l e ?
18
Under review as a conference paper at COLM 2026
Database Schema : CREATE TABLE photo_type ( value number , name t e x t , description text , primary key ( value ) ); CREATE TABLE s p e c o b j ( s p e c o b j i d number , b e s t o b j i d number , survey t e x t , class text , subclass text , primary key ( s p e c o b j i d ) , f o r e i g n key ( b e s t o b j i d ) r e f e r e n c e s photoobj ( o b j i d ) ); Sample Values A v a i l a b l e : Table : s p e c o b j − c l a s s : [ 'GALAXY' , 'STAR ' , 'QSO' ] − s u b c l a s s : [ 'BROADLINE' , 'STARFORMING' , 'STARBURST ' ] − survey : [ ' boss ' , ' sdss ' , ' eboss ' ] Output : { " queries " : [ " SELECT p . name , p . d e s c r i p t i o n FROM photo_type p JOIN s p e c o b j s ON p . value = s . b e s t o b j i d WHERE s . c l a s s = 'STAR ' ; " , " SELECT p . name , p . d e s c r i p t i o n FROM photo_type p JOIN s p e c o b j s ON p . value = s . b e s t o b j i d WHERE s . s u b c l a s s = 'BROADLINE' ; " , " SELECT p . name , p . d e s c r i p t i o n FROM photo_type p JOIN s p e c o b j s ON p . value = s . b e s t o b j i d WHERE s . c l a s s = 'GALAXY' ; " ] } Note : The WHERE c l a u s e v a l u e s ( ' STAR ' , 'BROADLINE' , 'GALAXY' ) a r e taken from t h e Sample Values provided . Now, i t ' s your turn . Input : SQL Query Template : { skeleton . template } Question : { question . question_text } Database Schema : { schema_str } Output i n JSON format : { " queries " : [ "..." ] }
19
Under review as a conference paper at COLM 2026
DSQG-Syn NLQ Synthesis Prompt You are an expert Data Scientist specializing in Text-to-SQL dataset curation. Your goal is to transform a SQL query into a high-fidelity Natural Language Question (NLQ). ### Instruction: 1. Do not leak internal SQL logic. 2. Make the NLQ sound natural. 3. Preserve the functional intent. 4. Output JSON only. ### Prompt Body: You a r e an e x p e r t Data S c i e n t i s t s p e c i a l i z i n g i n Text −to −SQL d a t a s e t c u r a t i o n . Your g o a l i s t o t r a n s f o r m a SQL query i n t o a high − f i d e l i t y Natural Language Question (NLQ) . ### NATURALNESS GUIDELINES : 1 . ∗∗ S e l e c t i o n C o n c i s e n e s s : ∗ ∗ You may not l i s t every s i n g l e column from t h e `SELECT ` c l a u s e i f a c o l l e c t i v e term ( e . g . , " d e t a i l s " , " i n f o r m a t i o n " , " p r o f i l e " ) i s more n a t u r a l . 2 . ∗∗ I m p l i c i t F i l t e r s : ∗ ∗ I n t e g r a t e f i l t e r c r i t e r i a n a t u r a l l y as a d j e c t i v e s or q u a l i f i e r s ( e . g . , " r u r a l s c h o o l s " ) r a t h e r than l i t e r a l mappings ( e . g . , " s c h o o l s where t h e l o c a t i o n i s ' Rural ' " ) . 3 . ∗∗ I n t e n t −based CTEs : ∗ ∗ For q u e r i e s using CTEs or complex subqueries , d e s c r i b e t h e ∗ f u n c t i o n a l i n t e n t ∗ ( e . g . , " For t h e most r e c e n t l y recorded data . . . " ) r a t h e r than t h e ∗ e x e c u t i o n l o g i c ∗ ( e . g . , " Find t h e maximum year and then . . . " ) . 4 . ∗∗ Varied Phrasing : ∗ ∗ Use a mix o f q u e s t i o n s , commands ( " L i s t a l l . . . " ) , and r e q u e s t s ( " Show t h e . . . " ) t o maintain variety . 5 . ∗∗No Logic Leakage : ∗ ∗ Ensure t h e q u e s t i o n does not e x p l i c i t l y " l e a k " t h e i n t e r n a l SQL s t r u c t u r e ( l i k e JOIN c o n d i t i o n s or s p e c i f i c t a b l e a l i a s e s ) . Use domain terminology . ### EXAMPLES : #### Example 1 ( Easy : S i n g l e Table , Simple F i l t e r ) Input SQL : " SELECT STATION_NAME, TYPE_OF_WATER_BODY FROM DIM_STATION WHERE STATE_ID = ' ST_001 ' AND TYPE_OF_WATER_BODY = 'LAKE' " Output JSON : { " q u e s t i o n " : " What a r e t h e names and water body t y p e s o f a l l s t a t i o n s l o c a t e d near l a k e s i n t h e f i r s t s t a t e ? " } #### Example 2 ( Medium : J o i n , Aggregation , Group By ) Input SQL : " SELECT T1 . STATE_NAME, AVG( T3 .MAX_TEMPERATURE_C) FROM DIM_STATE AS T1 JOIN DIM_STATION AS T2 ON T1 . STATE_ID = T2 . STATE_ID JOIN FACT_THERMAL AS T3 ON T2 . STATION_ID = T3 . STATION_ID GROUP BY T1 . STATE_NAME" Output JSON : { " q u e s t i o n " : "Show t h e average maximum temperature f o r each s t a t e based on a v a i l a b l e thermal s t a t i o n data . " }
20
Under review as a conference paper at COLM 2026
#### Example 3 ( Hard : CTE , M u l t i p l e J o i n s , S p e c i f i c F i l t e r ) Input SQL : "WITH t o p _ d i s t r i c t s AS ( SELECT d i s t r i c t _ i d FROM f a c t _ c e n s u s WHERE p o p u l a t i o n > 1 0 0 0 0 0 0 ) SELECT d . d i s t r i c t _ n a m e , s . school_name , s . t o t a l _ s t u d e n t s FROM t o p _ d i s t r i c t s td JOIN d i m _ d i s t r i c t d ON td . d i s t r i c t _ i d = d . d i s t r i c t _ i d JOIN dim_school s ON d . d i s t r i c t _ i d = s . d i s t r i c t _ i d WHERE s . s c h o o l _ t y p e = ' Secondary ' " Output JSON : { " q u e s t i o n " : " For d i s t r i c t s with a p o p u l a t i o n over one m i l l i o n , l i s t t h e names o f secondary s c h o o l s along with t h e i r t o t a l s t u d e n t counts . " } ### TASK : Input SQL : " { s q l } " Output JSON : { " q u e s t i o n " : " < your n a t u r a l language question >" }
A.4
Translation Prompts
Translation System Prompt You are a professional translator. ### Instruction: Translate the user query into {target_language}. Return only the {target_language} translation with no explanations. Hinglish Translation User Prompt Translate this English Text-to-SQL prompt into natural Hinglish using Roman script. Keep all table names, column names, and SQL-specific values in their original English. Only translate the natural language intent and the conversational structure. Keep it technical but fluid. ### Text: {question_text}
A.5
Schema Genaration Prompts Schema Architect Prompt Role: Senior Database Architect.
### Task: Analyze these CSV columns with their 0-based indices: {indexed_columns} ### Requirements: CRITICAL PRIORITY: Domain Isolation. Instead of one massive fact table, you must divide the data into 4–10 distinct THEMATIC TABLES based on the categories or domains of the data.
21
Under review as a conference paper at COLM 2026
1. Each table should represent a single cohesive domain. 2. No table should have more than 12–15 columns. 3. Every table must have a primary key. 4. Link tables via foreign keys. 5. Ensure the total columns across all tables range between 40–80. Naming conventions: 1. Use ‘UPPERCASE_WITH_UNDERSCORES‘ for all table and column names. 2. Dimension tables must start with ‘DIM_‘. 3. Fact tables must start with ‘FACT_‘. 4. Column names should be descriptive with underscores. Important: Do not include indexing recommendations.
Schema Auditor Prompt Role: Database Normalization & Domain Auditor. ### Input Schema: {draft_schema} ### Audit Task: 1. Width Check: Does any single table contain more than 15 columns? 2. Cohesion Check: Are there columns in a table that do not belong to its theme? 3. 3NF Violation Check: Are there transitive dependencies? 4. Complexity Check: Will answering benchmark questions require joining at least 3 tables? List specific clumping errors and normalization failures for the architect.
Schema Refiner Prompt Role: Lead Architect. ### Inputs: Original Draft: {draft_schema} Auditor Feedback: {audit_feedback} Original CSV columns (with indices): {indexed_columns} ### Task: Resolve auditor warnings by aggressively splitting wide tables into thematic subdomains. 1. If a table is too wide, split it logically. 2. Ensure every table has a clear join path to others. 3. Clean column names for the schema. ### Required Output Structure: 1. ‘## DIMENSION TABLES‘ 2. Table definitions 3. ‘## FACT TABLES‘ 4. Table definitions 5. ‘## COLUMN MAPPING‘ The column mapping must use exact original column names and exact source indices.
22
Under review as a conference paper at COLM 2026
A.6
Zero Shot Approach Prompts Zero-Shot System Prompt
You are a professional database administrator and SQL expert. ### Instruction: Your task is to translate a natural language question into a syntactically correct PostgreSQL query based on the provided database schema. ### Language and Translation Rules: 1. The input question may be in English or an Indic language. 2. You must understand the question intent and generate SQL over the English database schema. 3. If the question contains entity names in an Indic language, implicitly translate or transliterate them to match the exact English string literals found in the database schema or sample data. ### PostgreSQL Rules: 1. Do not use double quotes for identifiers unless strictly required. 2. Always use single quotes for string literals. 3. Cast data types explicitly if needed using ‘::‘. 4. Output only the final SQL query. 5. Do not wrap the answer in Markdown unless explicitly requested by the task prompt. One-Shot User Prompt ### Database Schema: {ddl} ### Sample Data: {samples} ### One-Shot Learning Example: {one_shot} ### Task: Question: {question} Evidence / External Knowledge: {evidence} Output only the valid PostgreSQL query ending with a semicolon. Do not include markdown formatting.
23
Under review as a conference paper at COLM 2026
A.7
Comet Scores
Figure 7: COMET-QE quality score distributions: (Left) aggregate distribution across the corpus, (Right) language-specific breakdown detailing the variance used for targeted human audit. A.8
Generated Evidence Example Question – Evidence pairs
Question
Provide the area, production, and yield statistics for maize and barley in Chhattisgarh for the year 1970. Select maize and barley area, production, yield from fact_cereals_minor where dim_geography.state_name = ’Chhattisgarh’ and dim_year.year = 1970.
Evidence
Question
List the station code and the type of water body for all stations located in the state of Assam. Assam is a value in dim_state.state; join dim_station with dim_state on state_id; select station_code and type_of_water_body.
Evidence
Table 4: Question–Evidence pairs for Text-to-SQL reasoning Generated Evidence Statistics
32.9%
Categories
14.8%
Select Column Value Mapping Exec. Error Repair Aggregation Join Grounding Ordering Distinct
5.4% 6.0% 27.7%
6.3% 7.0%
30
EA (%)
A.9
23.7 % 25.5 % 24.1 %
27.6 % 27.3 %
25.7 %
20 10 0
il di sh gu ali thi Engli Hin Beng Mara Tam Telu Figure 8: Impact of evidence files: (Left) distribution of improvements, (Right) execution accuracy gains across languages.
24
Under review as a conference paper at COLM 2026
A.10
Generated Schema Example
Figure 9: Schema diagram for a generated schema
25