Database Context Compression for Text-to-SQL on Real-World Large Databases Jingwen Liu1,2,3,* , Weibin Liao1,2,3,*,† , Xin Gao1,2,3 , Junfeng Zhao2,3 , Yasha Wang1,3,4, B 1
arXiv:2606.28601v1 [cs.DB] 26 Jun 2026
National Engineering Research Center of Software Engineering, Peking University, Beijing, China 2 School of Computer Science, Peking University, Beijing, China 3 Key Laboratory of High Confidence Software Technologies, Ministry of Education, Beijing, China 4 Peking University Information Technology Institute, Tianjin Binhai, China [email protected], [email protected]
Abstract—Recent progress on Text-to-SQL has been driven by stronger language models and richer prompting strategies, yet performance on real enterprise benchmarks such as Spider 2.0 and BIRD remains far below that on classical academic datasets. We argue that the dominant bottleneck on such benchmarks is no longer reasoning ability, but the way the database is presented to the model. Real databases contain wide tables with repeated audit columns, large families of homogeneous partitioned tables, opaque machine-generated identifiers whose meaning lives only in column descriptions, and long data dictionaries in which only a small, query-dependent fraction is actually relevant. Existing query-aware approaches—schema linking, broadly construed to include retrieval-style schema subsetting—attempt to filter this raw context, but operate on top of a representation that is simultaneously structurally redundant, semantically verbose and documentation-heavy. We re-frame the problem as one of database context compression: a query-agnostic, database-side rewrite of the schema, semantic descriptors and external documents into a higher-density representation. We formalize this rewrite as the S GCF (Support– Gain Component Factorization) principle, which uniformly explains four very different operators—repeated column-group extraction, isomorphic table templating, shared semantic-tag componentization, and question-relevant evidence purification— as instances of a single coverage objective on different information layers. Building on S GCF, we present D B CC, a two-phase database-side middleware that performs query-agnostic structural and semantic re-encoding offline, and lightweight queryaware evidence purification online. D B CC is model-agnostic and pipeline-agnostic, and can be inserted before schema linking or generation in any existing Text-to-SQL system. On Spider 2.0Snow and BIRD, D B CC reduces input tokens by up to two orders of magnitude (on the most challenging L ARGE bucket of Spider 2.0-Snow, 2.6 M → 34.7K tokens) while raising strictrecall on schema linking from 0% to 56.5% on that bucket under DeepSeek-V3.2 (and to 63.1% under Claude-Opus-4.7), and yields a 1.8–1.9% absolute end-to-end EX improvement when stacked on top of each of three recent Text-to-SQL systems. Our code is open-sourced at https://github.com/MrBlankness/ SchemaCompression. Index Terms—Text-to-SQL, large databases, context compression, schema linking, natural language interfaces to databases
I. I NTRODUCTION Natural-language interfaces to relational databases promise to lower the bar for accessing the data that increasingly * Equal contribution.
† Project lead.
B Corresponding author.
Schema Redundancy
Wide Tables and Repeated Column Groups
Homogeneous Partitioned Tables
Column-semantic Verbosity
Composite and Semi-structured Columns
Opaque Machine-generated Identifiers
External-document Irrelevance
Verbose and Unevenly Relevant External Documents
Fig. 1. The bottleneck of Text-to-SQL on real enterprise databases. A representative Spider 2.0 schema contains thousands of columns drawn from wide audit-heavy tables, large families of sharded partitions, opaque machinegenerated identifiers and lengthy data dictionaries; existing query-aware schema-linking pipelines must filter this raw context, but the input itself is structurally redundant, semantically verbose and documentation-heavy. D B CC takes the orthogonal path of rewriting the database once on the database side—yielding a compressed view D′ (q) = (S ′ , M ′ , E ′ (q)) that is then consumed by any downstream pipeline—instead of repeatedly filtering the raw D per query.
drives modern enterprises. The Text-to-SQL task, which translates a user question into an executable SQL query, is the workhorse of this vision [1]–[8]. Driven by large language models (LLMs), Text-to-SQL accuracy on classical academic benchmarks has reached impressive levels [9]–[14]. The picture changes sharply once these systems are deployed on real enterprise data. Benchmarks such as BIRD [15] and especially Spider 2.0 [16] expose a substantial gap: average databases contain on the order of 103 columns, queries involve long join chains, business semantics are often hidden in opaque identifiers, and the surrounding documentation may run to tens of pages per database. On such inputs, even the strongest LLMs degrade markedly. The natural reaction is to
attribute this degradation to insufficient model capacity, longer reasoning chains, or richer prompts. Surveys of the field [7], [8] have begun to suggest, however, that the bottleneck has shifted: the dominant source of error is no longer how the model reasons about the database, but how the database is presented to the model in the first place. Fig. 1 summarizes this shift: existing query-aware schema-linking pipelines are forced to filter a raw context that is itself structurally redundant, semantically verbose and documentation-heavy, whereas D B CC rewrites the database once on the database side and lets every downstream pipeline consume the resulting compact view. A. Why “larger” databases are not just larger Treating large databases as a quantitative scaling problem misses what makes them qualitatively different. Inspecting the databases shipped with Spider 2.0 [16] and BIRD [15] reveals that the growth of physical schema size is dominated by engineering artefacts rather than business semantics: • Wide tables and repeated column groups. For performance and historical reasons, business tables are often denormalized, and identical groups of audit, status or measurement columns are pasted across many tables. • Homogeneous partitioned tables. Time, region or version sharding produces large families of physical tables whose schemas are essentially identical to a common parent. • Composite and semi-structured columns. VARIANT, JSON, ARRAY and STRUCT columns hide the discriminative information inside nested attributes, where neither column name nor type alone is sufficient for matching. • Opaque machine-generated identifiers. Many column names are abbreviations, codes or auto-generated tokens whose business meaning lives only in external column descriptions. • Verbose and unevenly relevant external documents. Data dictionaries, metric definitions and business notes provide essential disambiguation, but each document is long, only a small fraction is relevant to any given question, and the rest acts as in-context noise. These five observations are not five independent problems. They cluster into three challenges, each defined by the layer of the database context at which low information density arises and by whether that density is intrinsic to the database or to the question. Challenge #1 (Schema redundancy). Observations 1–2 inflate the schema layer with content that is repeated across tables: identical groups of audit, status or measurement columns reappear in many wide tables, and isomorphic schemas reappear across families of sharded tables. The redundancy is structural and query-agnostic—a column group repeated in 30 tables is repeated regardless of who is asking. Challenge #2 (Column-semantic verbosity). Observations 3–4 inflate the column-semantics layer: composite and semi-structured columns force the system to inspect nested attributes that neither column name nor type alone disambiguates, while opaque machine-generated identifiers shift the
burden of meaning onto verbose external column descriptions. The cost paid per column is high, and—because similar concepts are described differently across tables—the same semantic content is paid for many times. Challenge #3 (External-document irrelevance). Observation 5 inflates the external-knowledge layer in a qualitatively different way: each individual document is internally nonredundant, but only a small, question-dependent fraction of any document is relevant to a given question; the remainder acts as in-context noise. Unlike Challenges #1–#2, the redundancy here cannot be removed once and for all—it depends on the question. The combined effect is that real database context is simultaneously too large to fit and too unfocused to use. Sec. IV returns to these three challenges and shows how each is addressed by a corresponding operator inside D B CC. B. The shift we propose: from query-aware filtering to queryagnostic re-encoding Existing approaches to large-database Text-to-SQL implicitly assume that the raw database representation is fundamentally adequate, and that the system’s job is to filter, rank or select within it. Schema-linking methods, broadly construed— ranging from finer-grained alignment over the original schema (R AT-SQL [9], R ESD SQL [10], B RIDGE [17], RSLSQL [18]) to retrieval-style subsetting (C RUSH 4SQL [19], L INK A LIGN [20], C ODE S [21], DB-E XPLORE [22])—all decide what to keep on top of an unchanged database; agentic methods such as C HESS [12], R E F O RCE [23], M AC SQL [24] and AUTO L INK [25] explore the database iteratively, but again over its raw form. None of them rewrites the database itself. We argue that this is precisely what should be rewritten. The redundancy in real schemas is structural and largely query-agnostic, and the verbosity of column descriptions and external documents is intrinsic to how these artefacts were produced rather than to the question. Treating these regularities as engineering noise that should be compressed at the database side, once and for all aligns with classical database thinking— one builds an index, a materialized view or a summary statistic so that every subsequent query benefits. The re-framing has three direct consequences: (i) it separates a query-agnostic offline stage from the lightweight, query-aware online stage of evidence purification; (ii) it makes the offline stage a reusable database-side asset whose cost amortizes over many queries; (iii) it is orthogonal to query-aware methods, providing them with a more compact, more focused input. C. Contributions This paper makes four contributions. A new framing. We identify database context compression as a first-class problem layer for Text-to-SQL on real databases, distinct from schema linking and prompt compression (Sec. III). • A unified principle. We introduce the S GCF principle (Support–Gain Component Factorization), a single •
coverage-style objective that uniformly captures structural, semantic and external-knowledge compression as instances of the same rewrite operator on different information layers (Sec. IV). • Concrete algorithms. We design and analyze four operators under S GCF: column-group factorization for wide tables (Sec. V), template hierarchies for homogeneous partitions (Sec. V), shared semantic-tag componentization for column descriptors (Sec. VI), and question-relevant evidence purification for external knowledge (Sec. VII). • A pluggable middleware. We package the operators into D B CC, a two-phase database-side middleware that integrates with any Text-to-SQL system—traditional pipelines, prompt-based LLMs and agentic flows—without modifying their core logic (Sec. VIII). The remainder of the paper is organized as follows. Section II relates D B CC to the most closely connected lines of work. Section III formalizes database context compression. Section IV presents the S GCF principle and the D B CC architecture. Sections V–VII describe the structural, semantic and external-knowledge operators. Section VIII discusses integration with downstream systems. Section IX reports the experimental evaluation. Section X concludes. II. R ELATED W ORK We position D B CC against three lines of work and explicitly mark the boundary at each line. A. Text-to-SQL with LLMs The transition from supervised parsers [9], [10], [17], [26]– [30] to LLM-based pipelines [11]–[14], [31]–[35] has changed the focus of the field from designing a stronger parser to organizing a more useful task context. Methods such as D IN SQL [11], C HESS [12] and C HASE -SQL [13] decompose the prompt or aggregate multiple reasoning paths; M AC SQL [24], R E F O RCE [23] and AUTO L INK [25] generalize this view to multi-agent or self-refining flows; generate-thenrank pipelines such as G AR [36] and M ETASQL [37] rerank an SQL-candidate pool—rather than schema candidates—against the input question. Surveys of LLM-era Text-to-SQL [7], [8], in-depth benchmarking studies [38], and benchmark papers such as Spider 2.0 [16] and BIRD [15] document a clear pattern: improvements at the model side saturate, while contextside issues—noise, redundancy and length—become the new dominant factor. D B CC sits squarely on the database side of this divide and is complementary to all of the above. B. Schema linking Schema linking—understood broadly as deciding which elements of the database are relevant to a question— spans two complementary granularities. Token-level alignment was first developed inside end-to-end parsers (R ATSQL [9], R ESD SQL [10], B RIDGE [17]) and later isolated as a standalone module in dedicated linkers (RSLSQL [18], L INK A LIGN [20], ISESL-SQL [39]); retrievalstyle approaches [19], [21], [22], [40] narrow a far larger
candidate space when the full schema cannot be fed in directly. Disambiguation frameworks such as CLEAR [41] target a different axis—resolving question-side ambiguity rather than schema-side redundancy—and are likewise complementary to D B CC. Both families are query-aware: they decide what is relevant given the user question, on top of an unchanged database representation. D B CC differs in two crucial ways. First, it is query-agnostic for its structural and semantic operators: it rewrites the database once, and the rewrite serves every subsequent query. Second, it is lossless with respect to schema content: a templated table or a factorized column group can always be expanded back to its original definition, so downstream linking does not lose any candidates. In our integration analysis (Sec. VIII) and ablations (Sec. IX) we therefore evaluate D B CC both standalone and stacked on top of representative schema-linking baselines. C. Prompt compression and schema description Generic prompt-compression methods such as LLML IN [42] and the Lost-in-the-Middle analysis [43] target token-level redundancy in arbitrary prompts. S CHEMONIC [44] is closest in spirit to our work: it generates succinct naturallanguage descriptions of database schemas to lower the cost of LLM prompting. A separate line of database-side adaptations targets specific deployment regimes—domain-specific finetuning frameworks such as F IN SQL [45], hybrid small/large LM compositions [46], and very-long-context LLM strategies [47]—all of which still consume the raw schema verbatim. D B CC differs in that we exploit structural regularities of the database (column-group repetition, table isomorphism, semantic-tag sharing) to perform a lossless rewrite, rather than a textual summarization or an enlarged context window. The two are complementary: S CHEMONIC-style summaries can be applied to the residual textual layer that survives our structural compression. GUA
III. P ROBLEM F ORMALIZATION This section formalizes database context compression as a database-side rewrite problem. A. Database context Let a database D be associated with three kinds of context information that a Text-to-SQL system may consume: • the schema S, comprising tables and their typed columns and key relationships; • the column-level semantics M , comprising naturallanguage column descriptions, types and sample values; • the external knowledge E, comprising data dictionaries, metric definitions and business documents. The raw database context is denoted D = (S, M, E). Given a question q, an end-to-end Text-to-SQL system Π produces a SQL query ŷ = Π q, D .
B. The compression problem A database compressor is a map C : D 7−→ D′ (q) = (S ′ , M ′ , E ′ (q)) that rewrites database context into a more compact representation D′ , possibly conditioned on q for the external-knowledge component. Let Cost(·) denote a measure of input cost— throughout this paper we use input-token count, which is also the dominant per-call billing unit for commercial LLM APIs— and Util(Π, D, ·) denote a measure of downstream utility (e.g., schema-linking recall or execution accuracy). The compressor is required to satisfy: Problem 1 (Database context compression): Find C such that for a target downstream system Π and a workload of questions {qi }, the compressed context simultaneously satisfies, in expectation over {qi }, Cost D′ ≪ Cost D (1) and Util Π, D′ , qi
≥ Util Π, D, qi .
(2)
Three properties of Problem 1 are worth emphasizing. (i) Asymmetric in q. The structural part S ′ and the semantic part M ′ are produced by query-agnostic rewrites: they depend only on D and can be precomputed and cached at the database side. The external-knowledge part E ′ is the only query-aware component and is computed online. This asymmetry is the basis for the two-phase architecture in Sec. IV. (ii) Lossless with respect to schema content. D B CC discards no base table or column. Compression is realized through structured references—shared components and template inheritance—that can be deterministically expanded back, so that downstream linking has access to the full candidate space whenever needed. (iii) Inequality in utility, not equality. Equation (2) permits strict improvement, Util(D′ ) ≥ Util(D), because compressing redundant or noisy context often improves downstream accuracy by removing distractors—a phenomenon consistent with the Lost-in-the-Middle effect [43]. IV. T HE D B CC F RAMEWORK This section shows how Problem 1 is solved uniformly across very different information layers. A. The S GCF principle The redundancies described in Sec. I-A—repeated column groups across wide tables, isomorphic families of partitioned tables, repeated semantic tags across column descriptions, and repeated business concepts scattered across external documents—all share the same structure. They are repeated subsets of information units carried by multiple objects. Definition 1 (Information layer): An information layer is a triple ⟨O, U, cost⟩ where O is a set of carrier objects (e.g., tables, columns, document chunks), U : O → 2V assigns to each carrier a set of information units drawn from a vocabulary V, and cost(·) is a per-unit cost function (e.g., tokens).
A component on an information layer is a subset X ⊆ V of information units shared by multiple carriers. Replacing every shared occurrence of X with a reference to a single component definition removes redundancy. Definition 2 (Support, width and gain): For a candidate component X ⊆ V on layer ⟨O, U, cost⟩, its support s(X) is the number of carriers that contain X, its width w(X) is the cardinality of X, and its gain is s(X) = {o ∈ O : X ⊆ U (o)} , gain(X) = s(X) · w(X) − {z } | redundant cost saved
w(X) = |X|, s(X) + w(X) , (3) | {z }
component definition + references
under the convention that each information unit and each reference token costs one unit (the absolute scale of gain thus differs from real tokenizer costs by a small constant factor, but the ranking of components is preserved). Proposition 1 (Unified compression objective): Let X be a family of admissible components subject to layer-specific feasibility constraints (e.g., minimum support, maximum width, no overlapping with already-selected components). The unified compression problem is to maximize the total gain X max gain(X), (4) X
X∈X
which is a coverage-style objective with a per-component reward s(X) · w(X) and a per-component definition penalty s(X) + w(X). D B CC solves it on each layer with a greedy heuristic that iteratively selects the highest-gain admissible component (Algorithms 1–3); we treat this as a layer-specific design choice rather than a worst-case approximation guarantee, since the penalty term breaks the standard non-negative monotone-submodular setup that would otherwise yield a (1−1/e) bound. What is novel is the proposition’s unifying role across very different layers. Every operator inside D B CC (Sec. V–VII) instantiates Equation (4) with a different carrier/informationunit pair: tables and column signatures (horizontal structural), table clusters and column sets (vertical structural), columns and keyword sets (semantic), document chunks and evidencespan sets (external knowledge). The three layers are in oneto-one correspondence with the challenges of Sec. I-A: Challenge #1 (schema redundancy) is targeted by the structural layer, with column-group factorization (Sec. V-A) handling the horizontal variant from wide tables and template hierarchy (Sec. V-B) the vertical variant from homogeneous partitions; Challenge #2 (column-semantic verbosity) is targeted by the semantic layer, where hierarchical keyword extraction normalizes per-column descriptions and shared semantic-tag componentization removes the resulting cross-column repetition (Sec. VI); Challenge #3 (external-document irrelevance) is intrinsically question-specific and is targeted by question-driven evidence purification (Sec. VII). The first two layers are queryagnostic and can be precomputed once per database; the third is query-aware and is computed per question—an asymmetry that motivates the two-phase architecture introduced next.
SGCF Principle: Support-Gain Component Factorization
𝑔𝑎𝑖𝑛 𝑋 = 𝑠 𝑋 ( 𝑤 𝑋 − (𝑠 𝑋 + 𝑤(𝑋)) Redundant cost saved Raw Database Schema
Schema 𝑆
…
Phase I: Offline · Query-Agnostic · Cached per Database Structural Compression
Component Factorization … Table A
…
Tables, Columns, Keys.. Semantics 𝑀
Template Hierarchy Table B Table A Table C
… Table B
Shared Component
Knowledge
+
Semantic Compression datetime
DB-Level
Column: ID • Description:… • Type: String • Samples:[123,124,..]
audit
Table-Level
mutation
Chunk-Level
Column A …
𝑆 (Structural)
Concept-to-element mappings. Metric definitions
…
Column B
Domain definitions
Shared Semantic-tag
Compressed View D! = (𝑆 ! , 𝑀! , 𝐸′(𝑞)) !
External Doc User Query
Value constraints
> Shared Semantic-tag Componentization Data Dictionary, Metric Defs
Evidence Purification
+
+
> Hierarchical Keyword Extraction
𝐸
Phase II: Online · QueryAware · per Question
Parent Template
…
Table C
Table
Descriptions, Types, Samples…
Component definition + References
!
𝑀 (Semantic)
𝐸′(𝑞) (Evidence)
Pre-linking
Schema Linking
Pre-generation
SQL Generation
Executable SQL
Fig. 2. Architecture of D B CC. Phase I (offline, query-agnostic, per database) rewrites the raw database context D = (S, M, E) into a compact view through two operators: a structural operator that turns wide tables and families of sharded tables into column-group factorizations and template hierarchies S → S ′ , and a semantic operator that turns verbose column descriptions into hierarchical keyword tags and shared-tag components M → M ′ . Phase I outputs a cached view (S ′ , M ′ ) that is reused by every subsequent query. Phase II (online, query-aware, per question) runs only when a question q arrives: the evidence-purification operator turns long external documents E into a small set of typed evidence atoms E ′ (q) aligned to the schema. The final output D′ (q) = (S ′ , M ′ , E ′ (q)) is consumed without modification by downstream Text-to-SQL systems—schema-linking pipelines, prompt-based LLMs or agentic flows. The two-phase split makes the dominant LLM-driven cost (semantic keyword extraction in Phase I) amortize across queries, while keeping per-query work to a thin question-conditioned step.
B. Architecture D B CC realizes S GCF in a two-phase architecture (Fig. 2). The split has four direct consequences: (C1) Amortization— the dominant LLM-driven step (semantic keyword extraction) lives in Phase I, so its cost is paid once per database; (C2) Reusability—Phase I produces a self-contained compressed view consumable by any downstream pipeline without modification; (C3) Statefulness vs. statelessness—Phase I is a stateful database asset (akin to an index or materialized view), Phase II is stateless and lightweight; (C4) Compositionality—because the three operators act on disjoint information layers, their gains are largely orthogonal, supporting clean ablation. A natural concern is whether structural, semantic and external-knowledge compression could cannibalize each other. Equation (3) makes the answer transparent: each operator selects components from a disjoint vocabulary V (column signatures vs. keyword tokens vs. evidence spans), so components selected by one operator cannot be selected by another. D B CC quantifies orthogonality empirically with the index Orth(A, B) = ∆(A∪B)−max(∆A, ∆B) / min(∆A, ∆B), where ∆ is the gain in downstream utility relative to the uncompressed baseline; positive values indicate that the two operators contribute additively (Sec. IX).
V. S TRUCTURAL C OMPRESSION The structural layer addresses redundancy in the schema itself. D B CC instantiates two operators that together cover the two dominant redundancy patterns of real schemas. A. Component factorization for wide tables Pattern: A column group is a set of column signatures X ⊆ VS that appears in many tables. Examples include audit columns (created_at, updated_at, updated_by), monetary status columns (amount, currency, status), and standardized location attributes. Repeating X in every table that contains it costs |X| · s(X) tokens; expressing X once as a shared component and inserting a reference in every supporting table costs |X| + s(X). Operator: For each candidate column group X, D B CC computes its support s(X), width w(X) = |X| and gain (3), and accepts X as a component iff s(X) ≥ τs , w(X) ≥ τw and gain(X) ≥ τg . To keep each table’s rendering readable when many components apply, we additionally enforce a pertable component quota K (the maximum number of references attached to any single table; we use K = 8 throughout). Algorithm: Naive enumeration of column subsets within a wide table is exponential. D B CC avoids this with an
Algorithm 1 Column-group factorization 1: ϕ(c) ← {T ∈ T : c ∈ T } for every column signature c. 2: G ← { XS = ϕ−1 (S) : S ⊆ T } (one candidate per equivalence class of ϕ; see below). 3: G ⋆ ← { X ∈ G : s(X) ≥ τs , w(X) ≥ τw , gain(X) ≥ τg }. 4: Sort G ⋆ by (gain(X), s(X), w(X)) in decreasing order. 5: C ← ∅. 6: for X ∈ G ⋆ in sorted order do 7: if X disjoint from every Y ∈ C and per-table quota K respected then 8: C ← C ∪{X}. 9: end if 10: end for S 11: Output: S ′ = { T 7→ (R(T ), U (T )\ X∈R(T ) X) }T ∈T , with R(T ) = {X ∈ C : X ⊆ U (T )}. inverted-index trick that exploits the structure of column groups. Why the inverted index: A wide table can have hundreds of columns, so subset enumeration is intractable. The key observation is that two columns sharing the same support set ϕ(c) = ϕ(c′ ) must co-occur in any column group, so the only candidates worth scoring are the equivalence classes of ϕ taken whole. We thus restrict G to one candidate per distinct support set rather than enumerating all subsets of each class— a deliberate heuristic pruning that may miss strict subsets but in practice does not, because gain (3) is monotonically maximised by the widest co-occurring group at a given support. Components are rewritten as references, not destroyed, so the operator is reversible: any downstream component can request the expanded form. B. Template hierarchy for homogeneous partitions Pattern: A template is a set of physical tables {T1 , . . . , Tk } whose column-set is identical. Time-, regionor version-sharded tables are templates by construction. A derived template is a template whose column-set is a superset of another template’s column-set, differing only by a small delta. Operator: D B CC compresses a template family by recording the parent template once, plus, for each derived template, only the delta. Inheritance gives a compact tree-of-tables representation that resembles object-oriented inheritance. Design rationale and complementarity with factorization: Two thresholds control the rendering: the minimum parent size τp ensures that the parent carries enough structural information to be informative, and the relative-delta threshold τr rejects derivations whose increment is too large to read as a small extension—in pilot studies, removing τr produced “inheritance” relations that were technically valid but unreadable, hurting downstream linking despite reducing tokens. Factorization handles horizontal redundancy (columns shared across tables) while template inheritance handles vertical redundancy (entire tables sharing the same shape); the two operate on different carriers and select from disjoint candidate spaces, and their interaction is evaluated in Sec. IX.
Algorithm 2 Template hierarchy construction 1: B ← { Bi }, where Bi groups physical tables sharing the same column signature Sig(Bi ). 2: Sort B by |Sig(·)| in increasing order. Process Bi in this order, so that any candidate parent Bj with Sig(Bj ) ⊆ Sig(Bi ) has already been admitted (or rejected) by the time Bi is visited; when no such Bj exists, Bi is left as a root. 3: For each Bi , define Par(Bi ) = arg
max
Bj : Sig(Bj )⊆Sig(Bi ), |Bj |≥τp
Sig(Bj ) ,
and ∆i = Sig(Bi )\Sig Par(Bi ) . 4: Accept the inheritance link Bi → Par(Bi ) iff gain(Bi ) ≥ τg and |∆i |/|Sig(Bi )| ≤ τr ; otherwise mark Bi as a root. 5: Output: forest F = {(Bi , Par(Bi ), ∆i )} of accepted links plus the roots. VI. S EMANTIC C OMPRESSION Even after structural compression, each column still carries verbose semantic context: a natural-language description, a type, and sample values. In real databases these descriptions are often highly redundant across columns—“timestamp of the last update”, “timestamp of the most recent update”, “lastupdate epoch milliseconds” are essentially the same tag. The semantic layer of D B CC compresses this redundancy in two stages. A. Hierarchical keyword extraction For each column ci with description di , type ti and sample values vi , D B CC produces a small set of normalized keywords Ki = Norm Kdesc (di , vi ) ∪ Ktype (ti ) . The keyword set is intended to act as a semantic tag rather than a paraphrase: it should support disambiguation in schema linking, not reproduce the natural-language description. To preserve cross-table consistency, D B CC extracts keywords hierarchically: at the database level when the input fits, then falling back to table level, then to chunk level, with explicit propagation of vocabulary between levels. Databaselevel extraction is essential: it ensures that synonymous concepts in different tables receive the same tag, which is the single largest source of subsequent component reuse. Normalized type tags such as type:variant and type:struct are exposed explicitly, so that downstream models can reason about composite columns without parsing nested type expressions. B. Shared semantic-tag componentization After hierarchical extraction, small subsets of keywords are observed to appear across many columns. The semanticcomponent operator applies S GCF (Eq. (3)) to the keyword layer, with carriers O = {ci } and information units drawn from the keyword vocabulary VM , so s(X) here counts the number of columns whose keyword set contains X. D B CC accepts X as a semantic component iff s(X) ≥ τs , w(X) ≥ τw
Algorithm 3 Frequency-priority semantic-component selection 1: Input: columns {ci } with keyword sets {Ki }; thresholds (τs , τw , τg ). 2: Enumerate candidate components X ← {X ⊆ Ki : |X| ≤ wmax }. 3: For each X ∈ X , compute s(X), w(X), gain(X) from (3). 4: Sort X by s(X), gain(X), w(X) in decreasing order. 5: C ← ∅. 6: for X ∈ X in sorted order do 7: if s(X) ≥ τs and w(X) ≥ τw and gain(X) ≥ τg and X disjoint from every Y ∈ C then 8: C ← C ∪ {X}; remove X from each affected Ki . 9: Recompute s(·), w(·), gain(·) for the candidates that overlap any modified Ki . 10: end if 11: end for 12: Output: component set C; per-column residuals Ui ← Ki \ S X∈C X. and gain(X) ≥ τg . Each column ci , originally tagged with Ki , is rewritten as a pair b i = (Ri , Ui ), K
Ri ⊆ C, Ui = Ki \
[
X,
X∈Ri
where Ri is the set of references from ci to accepted components and Ui is the residual keyword set carried inline by ci . Frequency-priority component selection: Unlike structural columns, single-column keyword sets are short, so candidate components X ⊆ Ki can be enumerated directly up to a width cap wmax . D B CC then selects components by a frequency-priority greedy sweep: sort all admissible candidates by support s(X) (with gain(X) and w(X) as tie-breakers), and accept each candidate in turn iff it (a) still satisfies the support/width/gain thresholds against the current state and (b) is disjoint on its keyword tokens from every previously accepted component. After each acceptance the residual keyword sets are refreshed and the remaining candidates are re-evaluated. The intuition is that a high-support component produces a thicker shared backbone for the components that follow, so paying its definition cost first amortizes best across the database; the loop terminates after at most O(|V|) acceptances. Algorithm 3 states the procedure formally. Why semantic compression cannot be replaced by structural compression: The two layers select components from disjoint vocabularies: structural components come from VS (column signatures), semantic components come from VM (keyword tokens). A column group such as (created_at, updated_at, status) is a structural component; a tag such as {datetime, audit, mutation} attached to the column updated_at is a semantic component. They co-exist in the compressed view, and the ablation in Sec. IX-E confirms additive utility.
VII. E XTERNAL -K NOWLEDGE C OMPRESSION External documents—data dictionaries, metric glossaries, domain notes—contain information that is essential for resolving ambiguous columns and complex business rules, but per question only a small fraction of any given document is relevant. A. Question-relevant evidence purification D B CC treats the external-knowledge layer as the only query-aware component. Given a question q and a document d, D B CC extracts evidence E ′ (q) = Purify(q, d), designed to support schema linking and SQL generation rather than to summarize the document as a whole. The purification procedure is a thin LLM-driven step that performs typed extraction on (q, d) followed by a deterministic schema-anchoring pass; the full prompt template, the JSON schema each atom must satisfy, and the rule-based fallback that triggers when the LLM emits a malformed atom are all released with our code (see the URL in the abstract). Each piece of evidence it returns is a typed atom bound to a specific decision the downstream system has to make when generating SQL. We use four atom types, which together form a minimal cover of where external documents intervene in SQL generation—which columns/tables are used, how they are combined, which values they are filtered by, and what question-side vocabulary even refers to: • Concept-to-element mappings. A natural-language concept in q is bound to a concrete schema element, driving schema linking. E.g. “net revenue” → REVENUE.NET_AMT. • Metric definitions. A business metric named in q is expanded into the expression that computes it, driving aggregation and projection. E.g. MoM growth = (cur − prev)/prev. • Value constraints. The encoding of a filter value is recovered so the literal in the WHERE clause is correct. E.g. STATUS is encoded as 1=created, 2=paid, 3=completed, 4=refunded, so the predicate “completed” must be written as STATUS=3, not STATUS=’completed’. • Domain definitions. A term in q that has no direct schema counterpart is expanded into a set of schema-visible elements. E.g. “EU” → {DE, FR, IT, ES, NL,...} over COUNTRY_CD. Why purification, not retrieval: A retrieval-only baseline returns text passages, which still need to be parsed in context and consume non-trivial tokens. Purification produces typed evidence atoms that are easier for downstream linking to consume. This is also the layer at which S GCF degenerates: the carriers (document chunks) are query-specific, so support is computed over the set of question-relevant chunks rather than all documents. Purification can be viewed as a Text-toSQL-specific instantiation of retrieval-augmented generation, with the crucial difference that purified targets are schemaaware—each evidence atom is anchored to a database element, making it directly usable by schema linking. D B CC can also
be combined with arbitrary retrievers (e.g., BM25 [48] or dense retrievers [49]) as a pre- or post-processing step. VIII. I NTEGRATION WITH T EXT- TO -SQL P IPELINES Because D B CC rewrites only the database side, it integrates with a wide spectrum of downstream systems through a single contract: replace the system’s view of D with the compressed view D′ . A. Insertion points Within a typical Text-to-SQL pipeline D B CC can be inserted at two points, mirroring the two stages of the pipeline itself—schema linking and SQL generation. The two insertion points address different bottlenecks of the downstream system, and D B CC contributes a different aspect of D′ to each. • Pre-linking (shrinks the candidate space and sharpens token-level alignment). The system must decide which database elements are relevant to the question. We consider schema linking broadly, covering both granularities: tokenlevel alignment over the full schema [9], [10], [17], [18] and retrieval-style subsetting that narrows a far larger candidate space [19]–[22]. D B CC contributes to both. At the size axis, replacing S with the structurally compressed S ′ collapses hundreds of sharded variants into one canonical instance and exposes each repeated audit/status block once instead of dozens of times; the linker’s input is smaller, and its recall ceiling rises because near-duplicate candidates that previously competed for the top-k slots are now merged. At the density axis, hierarchical semantic tags in M ′ replace verbose, inconsistently worded column descriptions, and purified evidence atoms in E ′ provide explicit NL→element anchors; the linker sees fewer, more discriminative cues per candidate. • Pre-generation (reduces prompt cost and agent loops). If the system directly prompts an LLM [11], [13], [14], [31]– [33] or runs an agentic flow [12], [23]–[25], D′ shortens the prompt, focuses attention on discriminative information, and—for agents—cuts the number of database-exploration steps needed before the model commits to a plan. The two insertion points compose: a system that does linking and LLM generation in series can ingest D′ (q) once and benefit at both stages, because S ′ , M ′ , E ′ (q) are produced as a single self-contained view. B. Cost model and minimal assumptions D B CC does not modify the parser, decoder, schema-linking module, agent controller or post-processor of the downstream system: the rewrite is reversible (any system can expand S ′ back to S), no additional supervision is needed (structural/semantic compression is unsupervised, evidence purification needs only the question), and no per-query online learning state is kept on the database side beyond the cached Phase I view. Let cI (D) be the offline cost of Phase I, cII the per-query cost of Phase II, ∆c the per-query token saving, and N the expected number of queries against D. D B CC is profitable iff
N · ∆c ≥ cI (D) + N · cII , so the break-even number of queries is N ∗ = cI (D) / (∆c − cII ) . N ∗ is small whenever ∆c ≫ cII , which is the regime our compressors target; we measure N ∗ empirically in Sec. IX. IX. E XPERIMENTS We design our study around five questions: (Q1) How much can D B CC compress real database context, and at what utility cost or gain for downstream schema linking? (Q2) Are the three operators truly orthogonal, and how do they compose? (Q3) Does D B CC transfer across different LLMs, different benchmarks, and different downstream pipelines, and is the offline phase economically amortizable? (Q4) What does the cost–utility trade-off look like—is there a sweet spot, and is the curve monotone? (Q5) Where does D B CC still fail, and which residual errors should the next iteration target? A. Setup Benchmarks: We use Spider 2.0-Snow [16] as our primary benchmark. It is a real-workflow benchmark on Snowflake with 152 enterprise databases, 547 questions and an average of 812.1 columns per database; the average gold SQL involves 161.8 tokens and 6.8 function calls. The columncount distribution is heavy-tailed: while the mean is dominated by a long tail of small databases, a substantial minority sits in the ≥ 104 regime that motivates this work. Following Sec. I-A, we partition the databases into three buckets by column count: S MALL (< 1,000 columns; 76 DBs), M EDIUM (1,000–10,000 columns; 53 DBs) and L ARGE (≥ 10,000 columns; 23 DBs). For cross-benchmark generalization we use the full BIRD [15] benchmark—all of its 95 databases and 1,534 development questions, with no column-count filtering. Models: We use DeepSeek-V3.2 as the primary LLM for both Phase I (semantic keyword extraction, evidence purification) and the downstream task (schema linking, SQL generation). We additionally evaluate cross-LLM robustness with GPT-4o and Claude-Opus-4.7. Tasks and metrics: The primary task is schema linking: given a database context and a question, the LLM must output the set of schema elements (tables and columns) needed to answer the question. All metrics below follow the conventions established in recent schema-linking work—in particular RSLSQL [18] and Apex-SQL [50]—so that our numbers are directly comparable to theirs. We report: • Strict Recall Rate (SRR): per-instance indicator of whether the prediction covers all gold schema elements. • Non-Strict Recall (NSR) and Non-Strict Precision (NSP): micro-level recall and precision over predicted vs. gold elements. • Token cost (Tok): average input tokens per request; reported in K (thousand) or M (million). • Execution Accuracy (EX): for end-to-end Text-to-SQL experiments, the fraction of generated SQL queries whose execution result matches the gold query.
Default D B CC configuration: Structural compression uses template hierarchy, semantic compression uses the frequency-priority component selection of Algorithm 3 with thresholds (τs , τw , τg ) = (2, 2, 1), and external-knowledge compression uses the question-driven evidence purification described in Sec. VII. B. Structural Compression We first evaluate the two structural operators in isolation. Table I reports schema-linking utility and end-to-end EX, against the uncompressed schema, on the three database buckets. The two informative axes are the input cost (Tok) and the resulting utility (SRR/NSR/NSP/EX); their non-monotone trade-off is examined in Sec. IX-J. The two operators capture complementary redundancy patterns and select from disjoint candidate spaces (single tables for factorization vs. table clusters for hierarchy). Component factorization is stronger on S MALL and L ARGE databases (where audit/status groups dominate), template hierarchy on M EDIUM (where sharded families dominate). On S MALL the gain is essentially in cost: tokens drop by ∼ 22% while SRR/NSR/NSP stay within 1–3% of the baseline. On M EDIUM, template hierarchy already dominates: tokens drop by 82.0% ↑ (166.0K → 29.9K) and SRR rises by 5.2% ↑, NSR by 14.1% ↑. On L ARGE the contrast is striking: the uncompressed schema does not fit and yields SRR = 0.0%, while template hierarchy reaches 47.8% SRR with a 96.1% ↑ token reduction—compression is a prerequisite for the model to attempt the task at all. Per-database-type analysis: Classifying the 152 databases by structural fingerprint into W IDE -H EAVY (max column-signature support smax ≥ 10 and table-cluster ratio < 0.3; 58 DBs), PARTITION -H EAVY (table-cluster ratio ≥ 0.3; 41 DBs) and M IXED (the remaining 53 DBs) confirms what the design predicts. Factorization wins on W IDE -H EAVY (SRR 52.1 vs. 49.0 for hierarchy; EX 15.4 vs. 13.7); hierarchy wins on PARTITION -H EAVY (SRR 53.6 vs. 46.2; EX 16.8 vs. 11.5); on M IXED neither subsumes the other and applying both is preferable (SRR 51.9, EX 15.1). Hence the two operators select components from disjoint candidate spaces and are jointly needed. C. Semantic Compression Table II reports schema-linking utility and end-to-end EX of semantic compression vs. the raw representation, on the same three buckets. Tokens drop by 46.5% on S MALL (9.6K → 5.1K) and by roughly 73% on both M EDIUM and L ARGE (166.0K → 43.6K, 2.6M → 712.4K), confirming that semantic redundancy scales with the number of columns rather than with the size of any individual description. Utility stays close to the raw baseline on S MALL (where it is already saturated) and rises on the larger buckets (SRR 44.8 → 49.0 on M EDIUM; 0.0 → 8.7 on L ARGE). Semantic-only compression is not sufficient to make L ARGE databases tractable in isolation; the full benefit emerges only when combined with structural compression (Sec. IX-E).
D. External-Knowledge Compression We evaluate evidence purification on the subset of Spider 2.0-Snow questions that reference an external document. Table III reports the resulting schema-linking utility and endto-end EX on the same subset. At the prompt level the token savings from purification are essentially negligible—229.2K → 228.1K input tokens— because the external document is only one of several prompt components and the rest (schema, semantic descriptors, incontext examples) dominates. Despite this, replacing raw documents with purified evidence improves SRR by 2.7% ↑, NSR by 14.7% ↑ and NSP by 7.2% ↑ on the documented subset, with a corresponding gain of 2.6% ↑ in end-to-end EX. This is the qualitative claim of Sec. VII made empirical: the value of external-knowledge compression is not in length reduction but in the focus shift—business rules, conceptto-element mappings and value constraints become explicit, schema-anchored evidence atoms rather than being scattered across long prose. E. Ablation and Orthogonality We now evaluate the three operators jointly with a clean 23 -style ablation, where each operator is independently turned on or off. Structural compression uses template hierarchy, semantic uses frequency-priority, and external knowledge uses purified evidence; nothing else changes between rows, so the differences are attributable to the operators themselves. Table IV summarizes all eight configurations on the three buckets. Cumulative gains: On M EDIUM databases, full D B CC improves SRR by 11.5% ↑ and NSR by 18.6% ↑ over the uncompressed baseline, while reducing tokens by 91.4% ↑ (166.0K → 14.3K). On L ARGE databases the SRR gap widens to 56.5% ↑—essentially the difference between not running at all and a working linker—together with a 98.7% ↑ token reduction. On S MALL databases the utility is already saturated by the baseline; here D B CC preserves SRR/NSR within 1%– 2% while still cutting tokens by 53.1% ↑, indicating that the operators are not actively harmful on inputs that did not need compression. Orthogonality: We measure orthogonality via the index Orth(A, B) = ∆(A∪B) − max(∆A, ∆B) / min(∆A, ∆B) in terms of SRR gain over the baseline. Computing on the M EDIUM bucket using Table IV: • ∆(S) = 5.2, ∆(Sem) = 4.2, ∆(EK) = 1.0. • Orth(S, Sem) = (10.4 − 5.2)/4.2 = 1.24, Orth(S, EK) = (6.2−5.2)/1.0 = 1.00, Orth(Sem, EK) = (3.1−4.2)/1.0 = −1.10. Two pairs are clearly additive (Orth > 0) and one pair (semantic+EK without structural) is mildly antagonistic, which is interpretable: both semantic and EK operators rely on a high-quality column-level alignment, which only structural compression can deliver on L ARGE schemas. This interpretation is consistent with the L ARGE row of Table IV, where structural compression is what unlocks SRR above 40%.
TABLE I S TRUCTURAL COMPRESSION ON S PIDER 2.0-S NOW: SCHEMA - LINKING UTILITY AND END - TO - END EX, VS . THE UNCOMPRESSED SCHEMA . LLM: D EEP S EEK -V3.2. SRR/NSR/NSP/EX IN %; TOKENS IN K OR M. B EST PER METRIC PER BUCKET IN BOLD , SECOND - BEST UNDERLINED . S MALL Method
SRR NSR NSP
M EDIUM Tok
EX SRR NSR NSP
L ARGE Tok
EX SRR NSR NSP
Tok
EX
Raw schema 61.1 84.3 78.6 9.6K 28.6 44.8 67.3 57.1 166.0K 9.4 0.0 2.9 50.0 2.6M 0.0 Comp. factor. 58.3 83.0 78.3 7.4K 29.5 49.0 78.0 48.4 33.8K 15.8 43.5 67.6 63.7 82.9K 10.7 Templ. hier. 60.6 83.8 75.5 7.5K 30.4 50.0 81.4 57.4 29.9K 17.2 47.8 69.5 71.6 100.7K 11.6 TABLE II S EMANTIC COMPRESSION ON S PIDER 2.0-S NOW: SCHEMA - LINKING UTILITY AND END - TO - END EX, VS . THE RAW SEMANTIC REPRESENTATION . LLM: D EEP S EEK -V3.2. SRR/NSR/NSP/EX IN %; TOKENS IN K OR M. B EST PER METRIC PER BUCKET IN BOLD . S MALL Method
SRR NSR NSP
M EDIUM Tok
EX SRR NSR NSP
L ARGE Tok
EX SRR NSR NSP
Raw semantic 61.1 84.3 78.6 9.6K 28.6 44.8 67.3 57.1 166.0K 9.4 Keyword comp. 60.6 85.0 74.8 5.1K 29.1 49.0 76.4 47.5 43.6K 14.2 TABLE III E XTERNAL - KNOWLEDGE COMPRESSION ON THE S PIDER 2.0-S NOW SUBSET WITH EXTERNAL KNOWLEDGE . LLM: D EEP S EEK -V3.2. “T OK ” IS THE PROMPT- LEVEL INPUT- TOKEN COUNT. B EST PER METRIC IN BOLD .
Method
SRR
NSR
NSP
Tok
EX
Raw documents Purified evidence
49.3 52.0
73.9 88.6
69.4 76.6
229.2K 228.1K
12.7 15.3
F. End-to-End Integration To check whether a database-side rewrite translates to end-to-end gains, we plug D B CC into three recent Text-toSQL systems—AutoLink [25], ReFoRCE [23] and ApexSQL [50]—without modifying their pipelines, decoders, or post-processing. We restrict to Spider 2.0-Snow databases with >5,000 columns, where context organization most aggressively dominates accuracy. Table V shows that on all three systems, simply replacing the raw context with the D B CC-compressed view improves EX (by 1.8% ↑, 1.8% ↑ and 1.9% ↑ respectively) while substantially reducing token consumption (57.0% ↑ for AutoLink, 73.5% ↑ for ReFoRCE and 59.6% ↑ for Apex-SQL). ApexSQL is by a wide margin the strongest baseline on raw inputs (43.7% EX, more than twice that of AutoLink), reflecting the effectiveness of its agentic exploration loop at narrowing the database context per turn; even at this much higher operating point, D B CC compounds with that loop and adds a measurable EX gain at roughly 40% of the token cost. The absolute EX gains are modest (≤ 2%) and, as expected, smaller than the schema-linking improvements of Sec. IX-E: end-to-end EX is influenced by SQL planning, dialect handling, error recovery and post-processing, all of which are downstream of database context. D B CC only changes the input these systems consume, yet the EX improvement is positive on every one of the three systems, suggesting that real Text-to-SQL stacks are bottlenecked by context quality even when they invest in agentic flows or self-refinement. G. Cross-LLM Robustness We evaluate D B CC under two additional commercial LLMs, GPT-4o and Claude-Opus-4.7, on the same Spi-
Tok EX
0.0 2.9 50.0 2.6M 0.0 8.7 13.3 60.9 712.4K 2.6
der 2.0-Snow setting (Table VI). Two patterns emerge across both models. (i) On S MALL databases, where the uncompressed prompt already fits and is comprehensible, the relative gain shrinks because larger models already handle redundancy gracefully. (ii) On M EDIUM and especially L ARGE databases, where the issue is whether the prompt fits at all, the gap widens—a 2.6M-token raw prompt remains beyond the input budget of every LLM we tested, so absolute SRR stays at 0 without compression and rises sharply once D B CC is plugged in. Claude-Opus-4.7 is the strongest of the three on raw inputs and remains the strongest under D B CC, but the relative improvement that D B CC contributes is essentially constant across LLMs, supporting our framing: D B CC addresses an input-side problem and its benefit is largely independent of downstream model capacity. H. Cross-Benchmark Generalization We evaluate D B CC on the full BIRD benchmark (Table VII), with no column-count filtering. We use GPT-4o as the LLM here (rather than DeepSeek-V3.2 as in the rest of the experiments) because GPT-4o is the model on which existing BIRD numbers are most commonly reported, making the Raw row directly comparable to the literature; the crossLLM evidence in Sec. IX-G confirms that the ∆-pattern is essentially LLM-agnostic. Although BIRD has shorter average column counts than Spider 2.0-Snow, it features richer business semantics and external knowledge, so we expect proportionally larger gains from semantic and EK compression and slightly smaller gains from structural compression. The qualitative pattern transfers: D B CC consistently improves all three utility metrics while reducing tokens. The relative SRR gain is smaller on BIRD (7.3% ↑ vs. 22.7% ↑ on Spider 2.0-Snow) because BIRD’s databases—most of which fall well below the column-count regime that motivates this work—leave less raw structural redundancy to exploit; the gain on BIRD is therefore driven mostly by the semantic and external-knowledge operators rather than by structural rewriting.
TABLE IV F INE - GRAINED ABLATION OF D B CC ON S PIDER 2.0-S NOW. LLM: D EEP S EEK -V3.2. C OLUMNS S/S EM /EK INDICATE WHETHER STRUCTURAL , SEMANTIC AND EXTERNAL - KNOWLEDGE OPERATORS ARE ON . T HE ROW (✗,✗,✗) IS THE UNCOMPRESSED BASELINE ; (✓,✓,✓) IS FULL D B CC. B EST PER METRIC PER BUCKET IN BOLD , SECOND - BEST UNDERLINED . S MALL
M EDIUM
S Sem EK SRR NSR NSP
Tok
EX SRR NSR NSP
✗ ✗ ✗ ✓ ✗ ✓ ✓ ✓
9.6K 9.4K 5.1K 7.5K 5.0K 7.4K 4.7K 4.5K
28.6 29.0 29.1 30.4 29.3 30.3 30.7 31.2
✗ ✗ ✓ ✗ ✓ ✗ ✓ ✓
✗ ✓ ✗ ✗ ✓ ✓ ✗ ✓
61.1 62.3 60.6 60.6 60.0 59.1 61.1 61.7
84.3 84.3 85.0 83.8 82.0 83.0 83.3 82.4
78.6 76.1 74.8 75.5 75.3 74.7 71.4 71.8
44.8 45.8 49.0 50.0 47.9 51.0 55.2 56.3
67.3 77.8 76.4 81.4 74.2 78.9 77.3 85.9
L ARGE Tok
57.1 166.0K 9.4 0.0 2.9 50.0 62.2 165.4K 9.9 4.3 4.3 60.0 47.5 43.6K 14.2 8.7 13.3 60.9 57.4 29.9K 17.2 47.8 69.5 71.6 48.7 43.1K 12.7 13.0 24.3 71.8 54.6 29.3K 14.6 43.5 75.7 69.1 58.1 14.9K 16.4 52.2 76.7 72.9 55.7 14.3K 17.2 56.5 78.6 73.3
TABLE V E ND - TO - END PLUG - IN INTEGRATION WITH THREE RECENT T EXT- TO -SQL SYSTEMS ON S PIDER 2.0-S NOW DATABASES WITH >5,000 COLUMNS . LLM: D EEP S EEK -V3.2. B EST WITHIN EACH SYSTEM IN BOLD .
DB context
AutoLink [25] AutoLink ReFoRCE [23] ReFoRCE Apex-SQL [50] Apex-SQL
Raw + D B CC Raw + D B CC Raw + D B CC
EX
Tok
19.6% 21.4% 16.1% 17.9% 43.7% 45.6%
61.0K 26.2K 16.6K 4.4K 48.3K 19.5K
I. Comparison with Schema Linking and Pruning Baselines We compare D B CC against three representative queryaware baselines: Crush4SQL [19], LinkAlign [20] and RSLSQL [18]. We also report stacked settings where D B CC’s rewritten schema is fed into the baseline as input, to test whether the methods are orthogonal in practice. Three observations follow. (i) Standalone D B CC already outperforms every query-aware baseline on every metric while using ∼ 5× fewer tokens, supporting our framing that database-side rewriting tackles a different bottleneck from query-side filtering. (ii) Stacking D B CC with a query-aware baseline yields further gains (up to 2.7% ↑ SRR over standalone D B CC), confirming that the two families of methods are orthogonal: query-agnostic compression provides a denser candidate space, on which query-aware schema linking can then operate more reliably. (iii) The token-cost gap remains: even the stacked variants stay below 20K tokens, two orders of magnitude smaller than the raw schema. J. Cost–Utility Pareto Frontier To map out the trade-off, we sweep the thresholds of each operator and plot input-token cost vs. SRR on the L ARGE bucket. The sampled points (Fig. 3) reveal a non-monotone curve: shrinking the prompt initially raises utility, peaks at Tok ≈ 34.7K for full D B CC (matching the row in Table IV), and only at extreme over-compression (Tok ≲ 10K) starts to drop, when discriminative columns are absorbed into toosmall components—a manifestation of the Lost-in-the-Middle effect [43].
70 60
Tok
EX
2.6M 0.0 2.6M 1.0 712.4K 2.6 100.7K 11.6 716.3K 3.4 100.6K 9.4 34.9K 11.0 34.7K 11.6
sweet spot Tok=35K, SRR=56.5%
full DBCC structural only semantic only
50
SRR (%)
System
EX SRR NSR NSP
40 30 20 10 0 103
102
101
Input tokens to the linker (K, log scale)
Fig. 3. Cost–utility Pareto frontier on L ARGE databases of Spider 2.0-Snow. LLM: DeepSeek-V3.2. Three threshold sweeps are shown: structural-only, semantic-only, and full D B CC. The horizontal axis is the input-token cost (log scale, smaller-is-better, hence inverted); the vertical axis is SRR. SRR rises non-monotonically as the prompt shrinks, peaks at Tok ≈ 34.7K for full D B CC, and drops at extreme over-compression.
based hierarchical keyword extraction (Sec. VI-A); the structural rewrite is combinatorial and negligible, and Phase II adds a small per-question cost. The break-even N ∗ is small: even on S MALL databases, ∼4 questions already amortise the offline cost; on M EDIUM/L ARGE databases N ∗ = 1, i.e. D B CC is netpositive after a single query, since ∆c scales with database size while cII does not. L. Failure-Mode Analysis We finally take stock of where D B CC fails. Inspecting the schema-linking errors of the full D B CC configuration on Spider 2.0-Snow yields four recurring failure modes, whose distribution is shown in Fig. 4.
15.9% (n=20) 32.5% 21.4% (n=41) (n=27) 30.2% (n=38)
Mode 1: Discriminative column absorbed into a shared component (Structural) Mode 2: Inconsistent keyword normalization across tables (Semantic) Mode 3: Missing or noisy external document (Knowledge) Mode 4: Composite column hidden inside template's (Structural)
K. Offline Cost and Amortization We finally check whether the offline cost of Phase I is justified (Table IX). The Phase I cost is dominated by LLM-
Fig. 4. Distribution of failure modes of full D B CC on Spider 2.0-Snow. LLM: DeepSeek-V3.2. Slices show the share of each failure mode within the schema-linking error subset; absolute case counts in parentheses.
TABLE VI C ROSS -LLM ROBUSTNESS OF D B CC ON S PIDER 2.0-S NOW. W ITHIN EACH LLM, BEST PER METRIC PER BUCKET IN BOLD ( HIGHER IS BETTER FOR SRR/EX, LOWER FOR T OK ).
LLM
Method
DeepSeek-V3.2 DeepSeek-V3.2 GPT-4o GPT-4o Claude-Opus-4.7 Claude-Opus-4.7
S MALL
Raw + D B CC Raw + D B CC Raw + D B CC
M EDIUM
SRR
Tok
EX SRR
61.1 61.7 65.4 65.8 67.2 67.6
9.6K 4.5K 9.6K 4.5K 9.6K 4.5K
28.6 31.2 32.1 33.5 33.8 35.1
TABLE VII C ROSS - BENCHMARK GENERALIZATION OF D B CC ON S PIDER 2.0-S NOW AND THE FULL BIRD DEVELOPMENT SET. LLM: GPT-4 O . W ITHIN EACH DATASET, BEST PER METRIC IN BOLD ( HIGHER IS BETTER FOR SRR/NSR/NSP/EX, LOWER FOR T OK ).
Dataset
Method
SRR NSR NSP
Spider 2.0-Snow Raw Spider 2.0-Snow + D B CC BIRD Raw BIRD + D B CC
38.2 60.9 53.2 60.5
56.6 81.0 70.4 77.1
Tok
EX
55.7 925K 14.6 67.4 18.0K 22.3 60.8 38.6K 53.6 63.9 12.4K 58.2
TABLE VIII C OMPARISON WITH QUERY- AWARE SCHEMA - LINKING BASELINES ON S PIDER 2.0-S NOW. LLM: GPT-4 O . B EST PER METRIC IN BOLD ( HIGHER IS BETTER FOR SRR/NSR/NSP/EX, LOWER FOR T OK ).
Method
SRR NSR NSP
Raw Crush4SQL [19] LinkAlign [20] RSL-SQL [18] D B CC
38.2 47.5 51.0 49.5 60.9
56.6 67.4 71.5 70.1 81.0
Tok
EX
55.7 925K 14.6 58.1 102K 17.9 60.4 86K 19.0 61.7 96K 18.6 67.4 18.0K 22.3
D B CC + Crush4SQL 62.0 82.4 68.9 19.4K 22.9 D B CC + LinkAlign 63.6 83.5 68.5 17.6K 23.6
Mode 1 arises when an audit-style column that happens to be discriminative for a given query is folded into a frequent component; the column is still recoverable by expansion, but the compressed view does not surface it as salient. Mode 2 reflects the limits of LLM-driven keyword extraction: synonyms can survive normalization at the table level but disagree across tables, hurting cross-table linking. Mode 3 is upstream of D B CC and is shared by any external-knowledge-aware system. Mode 4 is specific to the template hierarchy operator: when the parent template is large, the per-derived ∆ may include composite columns that are easy to overlook. M. Summary, limitations and future work We summarise the answers to the five questions of Sec. IX. (Q1) On real L ARGE databases D B CC cuts linker input by 98% and lifts strict recall from 0% to 56.5% (DeepSeekV3.2)/63.1% (Claude-Opus-4.7), with end-to-end EX on Spider 2.0-Snow rising from 14.6% to 22.3% under GPT-4o. (Q2) The three operators are largely orthogonal (Orth > 0 on two of three pairs; layer-specific failure modes). (Q3) The benefit transfers across LLMs, benchmarks, and three downstream systems, stacks additively with three schema linkers, and
Tok
L ARGE EX SRR
44.8 166K 9.4 56.3 14.3K 17.2 49.6 166K 11.8 60.1 14.3K 19.6 51.4 166K 12.6 62.3 14.3K 20.7
Tok
EX
0.0 2.6M 0.0 56.5 34.7K 11.6 0.0 2.6M 0.0 60.7 34.7K 13.4 0.0 2.6M 0.0 63.1 34.7K 14.5
TABLE IX O FFLINE COST AND AMORTIZATION OF D B CC ON S PIDER 2.0-S NOW. LLM: D EEP S EEK -V3.2. N ∗ = BREAK - EVEN NUMBER OF QUERIES PER DATABASE . “P-I T OK ” AND “P-I S ” ARE P HASE I TOKEN COST AND WALL - CLOCK TIME PER DATABASE ; “P-II T OK ” IS P HASE II COST PER QUESTION ; “∆/ Q ” IS PER - QUERY SAVING ON THE DOWNSTREAM PIPELINE . Scale S MALL M EDIUM L ARGE
P-I Tok
P-I s
P-II Tok
∆/q
N∗
14K 138K 2.0M
26 240 3,500
0.7K 1.3K 1.6K
4.6K 152K 2.57M
4 1 1
Phase I amortises after one query on medium/large databases. (Q4) The cost–utility curve is non-monotone with a sweet spot at Tok ≈ 34.7K. (Q5) Discriminative-column absorption and cross-table keyword inconsistency drive 62.7% of residual errors, motivating a query-aware extension as future work.
X. D ISCUSSION AND C ONCLUSION Discussion. D B CC suggests two takeaways for the Textto-SQL community. First, the bottleneck has moved to the database side: even the strongest LLMs cannot ingest a 2.6M-token schema, the Lost-in-the-Middle effect [43] makes verbosity actively hurtful, and the resulting ceiling can be raised without touching the model or the linker. Second, classical database thinking carries over to the LLM era: our offline/online split mirrors the indexes / materialised-views idiom and the empirical N ∗ = 1 on medium/large databases shows the idiom translates almost verbatim. Whether S GCF extends to other database-side context layers (constraints, lineage, access policies, query logs) is left as explicit future work rather than claimed here. Together, these takeaways argue that database-side context engineering deserves first-class status alongside prompt design and linker architecture, and the offline/online split makes the resulting compressors immediately deployable atop existing Text-to-SQL stacks without retraining. Conclusion. Database context compression, grounded in S GCF and realised in D B CC, raises strict recall on enterprisescale databases from zero to the mid-fifties / low-sixties (LLMdependent) while cutting input tokens by roughly two orders of magnitude, with benefits that compose additively over stronger LLMs and existing schema-linking pipelines.
R EFERENCES [1] F. Li and H. V. Jagadish, “Constructing an interactive natural language interface for relational databases,” in Proceedings of the VLDB Endowment, vol. 8, no. 1, 2014, pp. 73–84. [2] N. Yaghmazadeh, Y. Wang, I. Dillig, and T. Dillig, “SQLizer: Query synthesis from natural language,” Proc. ACM Program. Lang., vol. 1, no. OOPSLA, pp. 1–26, 2017. [3] T. Yu, R. Zhang, K. Yang, M. Yasunaga, D. Wang, Z. Li, J. Ma, I. Li, Q. Yao, S. Roman, Z. Zhang, and D. Radev, “Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task,” in Proc. EMNLP, 2018, pp. 3911–3921. [4] D. Saha, A. Floratou, K. Sankaranarayanan, U. F. Minhas, A. R. Mittal, and F. Özcan, “ATHENA: An ontology-driven system for natural language querying over relational data stores,” Proc. VLDB Endow., vol. 9, no. 12, pp. 1209–1220, 2016. [5] C. Baik, H. V. Jagadish, and Y. Li, “Bridging the semantic gap with SQL query logs in natural language interfaces to databases,” in Proc. IEEE Int. Conf. Data Engineering (ICDE), 2019, pp. 374–385. [6] C. Baik, Z. Jin, M. J. Cafarella, and H. V. Jagadish, “Duoquest: A dual-specification system for expressive SQL queries,” in Proc. ACM SIGMOD Int. Conf. Management of Data, 2020, pp. 2319–2329. [7] X. Liu, S. Shen, B. Li, P. Ma, R. Jiang, Y. Luo, Y. Zhang, J. Fan, G. Li, and N. Tang, “A survey of text-to-SQL in the era of LLMs: Where are we, and where are we going?” IEEE Trans. Knowl. Data Eng., 2025. [8] Z. Hong, Z. Yuan, Q. Zhang, H. Chen, J. Dong, F. Huang, and X. Huang, “Next-generation database interfaces: A survey of LLM-based text-toSQL,” IEEE Trans. Knowl. Data Eng., 2025. [9] B. Wang, R. Shin, X. Liu, O. Polozov, and M. Richardson, “RAT-SQL: Relation-aware schema encoding and linking for text-to-SQL parsers,” in Proc. ACL, 2020, pp. 7567–7578. [10] H. Li, J. Zhang, C. Li, and H. Chen, “RESDSQL: Decoupling schema linking and skeleton parsing for text-to-SQL,” in Proc. AAAI, vol. 37, no. 11, 2023, pp. 13 067–13 075. [11] M. Pourreza and D. Rafiei, “DIN-SQL: Decomposed in-context learning of text-to-SQL with self-correction,” Adv. Neural Inf. Process. Syst., vol. 36, pp. 36 339–36 348, 2023. [12] S. Talaei, M. Pourreza, Y.-C. Chang, A. Mirhoseini, and A. Saberi, “CHESS: Contextual harnessing for efficient SQL synthesis,” arXiv preprint arXiv:2405.16755, 2024. [13] M. Pourreza, H. Li, R. Sun, Y. Chung, S. Talaei et al., “CHASE-SQL: Multi-path reasoning and preference optimized candidate selection in text-to-SQL,” arXiv preprint arXiv:2410.01943, 2024. [14] D. Gao, H. Wang, Y. Li, X. Sun, Y. Qian, B. Ding, and J. Zhou, “Text-toSQL empowered by large language models: A benchmark evaluation,” Proc. VLDB Endow., vol. 17, no. 5, pp. 1132–1145, 2024. [15] J. Li, B. Hui, G. Qu, J. Yang, B. Li, B. Li, B. Wang, B. Qin, R. Geng, N. Huo et al., “Can LLM already serve as a database interface? A big bench for large-scale database grounded text-to-SQLs,” Adv. Neural Inf. Process. Syst., vol. 36, pp. 42 330–42 357, 2023. [16] F. Lei, J. Chen, Y. Ye et al., “Spider 2.0: Evaluating language models on real-world enterprise text-to-SQL workflows,” arXiv preprint arXiv:2411.07763, 2024. [17] X. V. Lin, R. Socher, and C. Xiong, “Bridging textual and tabular data for cross-domain text-to-SQL semantic parsing,” in Findings of EMNLP, 2020, pp. 4870–4888. [18] Z. Cao, Y. Zheng, Z. Fan, X. Zhang, W. Chen, and X. Bai, “RSLSQL: Robust schema linking in text-to-SQL generation,” arXiv preprint arXiv:2411.00073, 2024. [19] M. Kothyari, D. Dhingra, S. Sarawagi, and S. Chakrabarti, “CRUSH4SQL: Collective retrieval using schema hallucination for Text2SQL,” in Proc. EMNLP, 2023, pp. 14 054–14 066. [20] Y. Wang, P. Liu, and X. Yang, “LinkAlign: Scalable schema linking for real-world large-scale multi-database text-to-SQL,” in Proc. EMNLP, 2025, pp. 977–991. [21] H. Li, J. Zhang, H. Liu, J. Fan, X. Zhang, J. Zhu, R. Wei, H. Pan, C. Li, and H. Chen, “CodeS: Towards building open-source language models for text-to-SQL,” Proc. ACM Manag. Data, vol. 2, no. 3, pp. 1–28, 2024. [22] H. Ma, Y. Shen, H. Liu et al., “DB-Explore: Automated database exploration and instruction synthesis for text-to-SQL,” arXiv preprint arXiv:2503.04959, 2025.
[23] M. Deng, A. Ramachandran, C. Xu, L. Hu, Z. Yao, A. Datta, and H. Zhang, “ReFoRCE: A text-to-SQL agent with self-refinement, format restriction, and column exploration,” in ICLR Workshop on VerifAI, 2025. [24] B. Wang, C. Ren, J. Yang, X. Liang, J. Bai, Q.-W. Zhang, Z. Yan, and Z. Li, “MAC-SQL: A multi-agent collaborative framework for text-toSQL,” in Proc. COLING, 2025, pp. 540–557. [25] Z. Wang, Y. Zheng, Z. Cao et al., “AutoLink: Autonomous schema exploration and expansion for scalable schema linking in text-to-SQL at scale,” in Proc. AAAI Conf. Artificial Intelligence, 2026, to appear. [26] R. Cao, L. Chen, Z. Chen, Y. Zhao, S. Zhu, and K. Yu, “LGESQL: Line graph enhanced text-to-SQL model with mixed local and non-local relations,” in Proc. ACL-IJCNLP, 2021, pp. 2541–2555. [27] T. Yu, C.-S. Wu, X. V. Lin, B. Wang, Y. C. Tan, X. Yang, D. Radev, R. Socher, and C. Xiong, “GraPPa: Grammar-augmented pre-training for table semantic parsing,” in Proc. ICLR, 2021. [28] X. Deng, A. H. Awadallah, C. Meek, O. Polozov, H. Sun, and M. Richardson, “Structure-grounded pretraining for text-to-SQL,” in Proc. NAACL, 2021, pp. 1337–1350. [29] T. Scholak, N. Schucher, and D. Bahdanau, “PICARD: Parsing incrementally for constrained auto-regressive decoding from language models,” in Proc. EMNLP, 2021, pp. 9895–9901. [30] J. Guo, Z. Zhan, Y. Gao, Y. Xiao, J.-G. Lou, T. Liu, and D. Zhang, “Towards complex text-to-SQL in cross-domain database with intermediate representation,” in Proc. ACL, 2019, pp. 4524–4535. [31] X. Dong, C. Zhang, Y. Ge, Y. Mao, Y. Gao, J. Lin, D. Lou et al., “C3: Zero-shot text-to-SQL with ChatGPT,” arXiv preprint arXiv:2307.07306, 2023. [32] H. Zhang, R. Cao, L. Chen, H. Xu, and K. Yu, “ACT-SQL: In-context learning for text-to-SQL with automatically-generated chain-of-thought,” arXiv preprint arXiv:2310.17342, 2023. [33] Z. Li, X. Wang, J. Zhao, S. Yang et al., “PET-SQL: A promptenhanced two-stage text-to-SQL framework with cross-consistency,” arXiv preprint arXiv:2403.09732, 2024. [34] T. Ren, Y. Fan, Z. He, R. Huang, J. Dai, C. Huang, Y. Jing, K. Zhang, Y. Yang, and X. S. Wang, “PURPLE: Making a large language model a better SQL writer,” in Proc. IEEE Int. Conf. Data Engineering (ICDE), 2024, pp. 15–28. [35] X. Li, Q. Cai, Y. Shu, C. Guo, and B. Yang, “AID-SQL: Adaptive in-context learning of text-to-SQL with difficulty-aware instruction and retrieval-augmented generation,” in Proc. IEEE Int. Conf. Data Engineering (ICDE), 2025, pp. 3945–3957. [36] Y. Fan, Z. He, T. Ren, D. Guo, L. Chen, R. Zhu, G. Chen, Y. Jing, K. Zhang, and X. S. Wang, “Gar: A generate-and-rank approach for natural language to SQL translation,” in Proc. IEEE Int. Conf. Data Engineering (ICDE), 2023, pp. 110–122. [37] Y. Fan, Z. He, T. Ren, C. Huang, Y. Jing, K. Zhang, and X. S. Wang, “Metasql: A generate-then-rank framework for natural language to SQL translation,” in Proc. IEEE Int. Conf. Data Engineering (ICDE), 2024, pp. 1765–1778. [38] O. Gkini, T. Belmpas, G. Koutrika, and Y. E. Ioannidis, “An in-depth benchmarking of text-to-SQL systems,” in Proc. ACM SIGMOD Int. Conf. Management of Data, 2021, pp. 632–644. [39] A. Liu, X. Hu, L. Lin, and L. Wen, “Semantic enhanced text-to-SQL parsing via iteratively learning schema linking graph,” in Proc. ACM SIGKDD Conf. Knowledge Discovery and Data Mining, 2022, pp. 1021– 1030. [40] Y. Zhang, A. Floratou, J. Cahoon, S. Krishnan, A. C. Müller, D. Banda, F. Psallidas, and J. M. Patel, “Schema matching using pre-trained language models,” in Proc. IEEE Int. Conf. Data Engineering (ICDE), 2023, pp. 1558–1571. [41] M. Zhang, K. Ma, L. Xu, K. Zhang, Y. Peng, and R. Jin, “CLEAR: A parser-independent disambiguation framework for NL2SQL,” in Proc. IEEE Int. Conf. Data Engineering (ICDE), 2025, pp. 1–14. [42] H. Jiang, Q. Wu, C.-Y. Lin, Y. Yang, and L. Qiu, “LLMLingua: Compressing prompts for accelerated inference of large language models,” in Proc. EMNLP, 2023, pp. 13 358–13 376. [43] N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang, “Lost in the middle: How language models use long contexts,” Trans. Assoc. Comput. Linguist., vol. 12, pp. 157–173, 2024. [44] I. Trummer, “Generating succinct descriptions of database schemata for cost-efficient prompting of large language models,” Proc. VLDB Endow., vol. 17, no. 11, 2024.
[45] C. Zhang, Y. Mao, Y. Fan, Y. Mi, Y. Gao, L. Chen, D. Lou, and J. Lin, “FinSQL: Model-agnostic LLMs-based text-to-SQL framework for financial analysis,” in Companion of the ACM SIGMOD Int. Conf. Management of Data, 2024, pp. 93–105. [46] J. Fan, Z. Gu, S. Zhang, Y. Zhang, Z. Chen, L. Cao, G. Li, S. Madden, X. Du, and N. Tang, “Combining small language models and large language models for zero-shot NL2SQL,” Proc. VLDB Endow., vol. 17, no. 11, pp. 2750–2763, 2024. [47] Y. Chung, G. T. Kakkar, Y. Gan, B. Milne, and F. Ozcan, “Is long context all you need? Leveraging LLM’s extended context for NL2SQL,” Proc. VLDB Endow., vol. 18, no. 8, pp. 2735–2747, 2025. [48] S. Robertson and H. Zaragoza, The Probabilistic Relevance Framework: BM25 and Beyond. Now Publishers Inc., 2009, vol. 4. [49] N. Reimers and I. Gurevych, “Sentence-BERT: Sentence embeddings using Siamese BERT-networks,” in Proc. EMNLP-IJCNLP, 2019, pp. 3982–3992. [50] B. Cao, W. Liao, Y. Sun, D. Fang, H. Li, and W. Lam, “APEX-SQL: Talking to the data via agentic exploration for Text-to-SQL,” in Proc. ACM SIGKDD Conf. Knowledge Discovery and Data Mining (KDD), 2026.