Beyond Tables: Doc2DB-Bench for Relationally Faithful Document-to-Database Construction

Beyond Tables: Doc2DB-Bench for Relationally Faithful Document-to-Database Construction Zhuowen Liang1 , Zhengxuan Zhang1 , Jiayang Wang1 , Jiazhuo Chen1 , Nan Tang1∗ 1 The Hong Kong University of Science and Technology (Guangzhou)

arXiv:2608.08459v1 [cs.CL] 9 Aug 2026

Abstract Practical AI systems increasingly need to turn long, heterogeneous documents into queryable relational databases, not isolated spreadsheets. In domains such as finance, healthcare, education, transportation, and enterprise operations, downstream workflows rely on normalized schemas, entity identities, keys, cross-table relationships, and integrity constraints for analytics, compliance, auditing, and SQL-backed decision making. Existing Document-to-Table benchmarks are insufficient for this setting: flattening evidence into single tables can duplicate entities, obscure many-to-many relationships, create sparse records, and avoid testing whether extracted facts form a valid database instance. This creates an urgent need to evaluate document understanding as database construction rather than field extraction. We introduce Doc2DB-Bench, a benchmark for Document-toDatabase construction, containing 203 long-document instances across 42 schemas and seven domain groups, with 117 entity tables, 132 relationship tables, 7,341 rows, and 41,935 cells. Built through a controllable DB-to-Doc synthesis pipeline and organized by a taxonomy of intra-table extraction and inter-table reasoning, the generated documents undergo authenticity verification, proving indistinguishable from real-world references. Doc2DB-Bench thus provides a testbed for reliable, auditable, and relationally faithful LLM-based data systems. The benchmark is publicly available at https://github.com/SetonLiang/Doc2DB-Bench

1

Introduction

Modern organizations rely on documents as the primary carrier of operational knowledge, from clinical notes and commercial contracts to financial reports and enterprise records [30, 6, 22]. Yet downstream applications rarely consume free-form text directly: analytics pipelines, Business Intelligence dashboards, compliance workflows, and SQL-backed systems require structured, queryable, and auditable data [28, 48, 46]. As large language models become increasingly capable of processing long and heterogeneous documents, converting document evidence into reliable structured data has become a central goal of information extraction [43, 26, 25]. Document-to-Table and Its Limitations. Most existing benchmarks study this problem under the Document-to-Table (Doc2Table) setting, where systems extract fields or populate a single flattened table [51, 9, 19, 47]. This setting is useful for isolated record extraction, but it is insufficient when downstream workflows require relational databases. Flattening multi-entity evidence can duplicate entities, obscure many-to-many relationships, introduce sparse records with excessive NULL values, and avoid testing whether extracted facts satisfy schema-level constraints. Example 1 Consider a financial report in Fig. 1(a) describing multiple companies, their cash positions across fiscal years, and investment relationships among them. The target schema in Fig. 1(b) contains an entity table Company and a self-referencing relationship table Hold, where company identifiers are system-generated keys and do not appear in the text. ∗ Corresponding author: Nan Tang (E-mail: nantang@hkust-gz.edu.cn)

ataMosaic, a document-to-database y mediates between extraction and atabase schema and constraints, a s entity and relationship extraction and targeted re-extraction within a p. By systematically resolving docut violations, DataMosaic incremensemantically consistent databases. , veri!ers and repair operators, exhow that DataMosaic substantially nd improves database-level accuracy

g1 , Jiazhuo Chen1 , Haixun Wang2 , Nan raction Meets Relational Semantics.

r artifacts have been made available at taMosaic.

ganizations, analytics, compliance, y operate over relational databases, ained model outputs. Yet much of ins embedded in unstructured docgap is challenging because docuocal—operating over isolated snipl databases enforce global semantics egrity constraints. This fundamental extraction, even when powered by abases that are consistent, auditable,

ment-to-database (Doc2DB) conollection 𝐿 , a target schema 𝑀, and

e Commons BY-NC-ND 4.0 International org/licenses/by-nc-nd/4.0/ to view a copy of overed by this license, obtain permission by eld by the owner/author(s). Publication rights

Vol. 14, No. 1 ISSN 2150-8097.

Treating Doc2DB as a pure extraction problem collapses these requirements into a single local decision, obscuring the role of database semantics [39].

investor_cid

investee_cid

1

2

2

1

valid but semantically inconsistent values. (3) Incomplete entity extraction. Entity information may b ing or partial when mentions are brief or indirect.

Error Patterns in Relationship Extraction. Relationship tion is harder, as it requires enforcing directionality, busines and semantics that extend beyond local textual evidence. (1) Hallucinated relationship extraction. Extractors may struct relationships that violate target semantics, such as sp reciprocal that contradictError no-reciprocity constraints. cid name !scal_year cash_beginlinks cash_end Patterns in Entity Extraction entity table is often treated as schema 1 Arena Tech 2023 (2)9,867,000 Missing11,200,000 relationship extraction. Extractors may also brittle in practice and prone to system 18,524relationships that Error are only implicitly cid2 name cash_begin cash_end Alpha Holdings !scal_year 2024 construct 32,368,000 Patterns in Entity supported Extraction. 18,524,000 (1) Hallucinated entity extraction. E entity table is often treated as schema 1 Arena Tech 2023 as transitive 9,867,000 11,200,000 3 Coyni Corp 2023 4,544,000 links, 4,062,000re"ecting the thatlimits shouldof notlocal exist,reasoning. including dup

E#$%&’( 1. Consider a !nancial report describing multiple com1 3 panies, their cash positions across !scal years, and investment rela2 3 tionships among them. The target schema consists of an entity table Company and a self-referencing relationship table Hold, primary (d) with Extracted Hold relationships: red (spurious), orange (missing). keys that are system-generated and do not appear in the text. Blue denotes entity mentions; purple denotes attribute values; red underlined values indicate sources of errors Arena Tech reported solid performance in !scal year 2023, with cash increasing from 9,867,000 at the beginning of the year to 11,200,000 at year end. Alpha Holdings !led its annual report for 2024. The report states that cash began at 18,524 (in thousand) and ended at 32,368,000. Alpha Holdings is a long-term investor with a signi!cant stake in Arena Tech. According to the company overview, Arena Tech is among the core holdings associated with Alpha Holdings. The report further notes that Arena Tech invested in Coyni Corp. Regarding Coyni Corp, the report records that its cash position declined from 4,544,000 to 4,062,000 during !scal year 2023. Elsewhere, Coyni Inc is mentioned as an a"liate company, but no corresponding !nancial !gures are reported.

! Entity Table: – Company(cid, name, fiscal_year, cash_begin, cash_end) – Key generation: cid GENERATED ALWAYS IDENTITY. ! Relationship Table: – Hold(investor_id, investee_id) – Foreign keys: investor_id, investee_id → Company.cid ! Integrity Constraints & Business Rules: 𝐿 1 : No mutual investment: ¬( Hold (𝑀, 𝑁 ) ↑ Hold (𝑁, 𝑀) ) 𝐿 2 : Cash sanity bound: 𝑂𝑀𝑃𝑄 _𝑅𝑆𝑇/𝑂𝑀𝑃𝑄 _𝑁𝑅𝑈𝑉𝑆 < 10 𝐿 3 : Recursive investment: Hold (𝑀, 𝑁 ) ↑ Hold (𝑁, 𝑂 ) ↓ Hold (𝑀, 𝑂 )

AS

brittle in practice and prone to system

18,524

solved aliasing and spurious entities i NULL 32,368,000 NULL 18,524,000 Why Doc2DB Is Fundamentally Hard. These examples (1) Hallucinated entity extraction. E (2) Attribute misinterpretation. Eve that should notextraction exist, including dupl that Doc2DB atviolate scale rect, extracted attributes may i (c) Extracted Company table: is not merely “information solved aliasing and spurious entities in 4 Coyni Inc 2023 full-"edged NULL database NULL construction characterized b ple byproblem ignoring units or temporal qua red (error); blue (correct). (2) Attribute misinterpretation. Even investor_cid investee_cid valid but semantically inconsistent va intertwined (c) Extracted Company table: red (error); blue challenges. (correct). rect, extracted attributes may violate in 24

Alpha CoyniHoldings Inc

2024 2023

3 Extracted Coyni CorpCompany table: 2023 red (error); 4,544,000 4,062,000 (c) blue (correct).

1

2

investor_cid 2

investee_cid 1

(3) Incomplete entity extraction. En

plemismatch. by ignoring units or temporal quaw Challenge 1: Document–schema Documents are ing or partial when mentions are brie valid but semantically inconsistent va as narratives or event sequences, whereas schemas are norm Error Patterns in Relationship Ext 11 23 (3) Incomplete entity extraction. Ent around entities, keys, and relationships. A itsingle paragrap tionorispartial harder,when as requires enforcing ing mentions are brie 22 13 and asemantics extend beyond loc populate multiple tables, while single that tuple may be Ext sca Error in Relationship 1 3 (d) Extracted Hold relationships: red (spurious), orange (missing). (1) Patterns Hallucinated relationship extrac across documents, making just extraction tion is harder, as brittle. it requires enforcing

2 3 struct relationships that violate targe and semantics that extend beyond loc reciprocal links that contradict no-re Challenge Things, not strings. Databases operate on entiti (d)!Extracted Hold relationships: red (spurious), 2: orange (missing). Entity(d) Table: (1) Hallucinated relationship extract Extracted Hold relationships: Missing relationship extraction. Company(cid, name, fiscal_year,keys, whereas documents provide(2) The goal to construct a relational database instance under a such (a) Aisfinancial report (b) Database schema definition Constructing a database instance requires –aligning extracted surface mentions. Sin structonly relationships that violate target red (spurious), orange (missing) cash_begin, cash_end) construct relationships that are only given target schema, consisting of an entity table Company andwith a selfevidence schema-level semantics. reciprocal links to thatentities contradictisno-rec are absent from text, mapping mentions inhe – Key generation: cid GENERATED ALWAYS AS as transitive links, re"ecting the limit ! Entity Table: IDENTITY. (2) Missingfragments relationship extraction. referencing relationship table Hold (Figure ??(c)). Crucially, company – Company(cid, name, fiscal_year,uncertain; forcing early key assignment facts and (1) Entity extraction. Figure ??(a) shows the entity table produced Why Doc2DB Is Fundamentally cash_begin, cash_end) construct relationships that are onlyH primary keys (cid) are system-generated surrogate identi!ers and ! Relationship Table: key- and join-consistent extraction brittle. – Key generation: cid GENERATED ALWAYS AS that Doc2DB is notre"ecting merely “informat by an LLM extractor. While many company mentions and attribute as transitive links, the limit – Hold(investor_id, investee_id) do not appear in the source text. IDENTITY. – Foreign keys: full-"edged database construction pr Challenge 3: No one-shot guarantees. Extraction operates ove values are correctly extracted, the result violates basic data semantics. Why Doc2DB Is Fundamentally H investor_id, investee_id → Company.cid intertwined challenges. ! Relationship Table: that Doc2DB and is notbusiness merely “informati Con!icting evidence leads to incorrect assignments, such as unit misrules r – Hold(investor_id, investee_id) snippets, whereas integrity constraints Challenge 1: Document–schema mism ! Integrity Constraints & Business Rules: – Foreign keys: full-"edged database construction pro match for Alpha Holdings w.r.t. 𝐿 2 . Moreover, entities mentioned only → Company.cid global veri"cation over the entire database instance. LLMs a 𝐿 1 :investor_id, No mutual investee_id investment: as narratives or event sequences, whe intertwined challenges. ¬( Hold (𝑀, 𝑁 ) ↑ Hold (𝑁, 𝑀) ) partially, such as Coyni Inc, are materialized as separate tuples with ditional IE pipelines can extract fragments of structure, bu around entities, keys, and relationsh 𝐿 2 : Cash sanity bound: Challenge 1: Document–schema misma ! Integrity Constraints & Business Rules: 𝑂𝑀𝑃𝑄 _ 𝑅𝑆𝑇/𝑂𝑀𝑃𝑄 _ 𝑁𝑅𝑈𝑉𝑆 < 10 populate tables, while a si missing attributes, introducing both incompleteness and duplication. provide no guarantees of integrity or multiple consistency. 𝐿𝐿1 : : No mutual investment: as narratives or event sequences, whe 3 Recursive investment: across documents, making just extrac ¬( Hold ) ↑Hold Hold Hold (𝑀,(𝑀, 𝑁 )𝑁 ↑ (𝑁,(𝑁, 𝑂 )𝑀)↓) Hold (𝑀, 𝑂 ) Taken together, these challenges tell uskeys, that Doc2DB re around entities, and relationshi (2) Relationship extraction. Figure ??(b) shows output of a 𝐿 2 : the Cash sanity bound: Challenge 2: Things, not strings. Datab 𝑂𝑀𝑃𝑄 _𝑅𝑆𝑇/𝑂𝑀𝑃𝑄 _𝑁𝑅𝑈𝑉𝑆 < 10 populate multiple tables, while a sin system-level support that goes beyond single-pass extractio straightforward relationship extractor based on surface-level textual 𝐿Constructing such a database instance requires aligning extracted 3 : Recursive investment: keys, whereas documents provide only across documents, making just extrac Hold (𝑀, 𝑁 ) ↑ Hold (𝑁, 𝑂 ) ↓ Hold (𝑀, 𝑂 ) evidence with schema-level are absent from text, mapping ment cues. The extracted Hold table reveals two fundamental failures. First, semantics. Contributions. We present DataMosaic, a Doc2DB system Challenge 2: Things, not strings. Databf uncertain; forcing early key assignme (1)Constructing Entity extraction. Figure ??(a) shows the entity table extracted produced contradictory phrasing induces a spurious mutual investment structing relational databases keys, from documents, with theonly fol such a(e.g., database instance requires aligning whereas documents provide keyand join-consistent extraction b by that an LLM extractor. Whilesemantics. many company mentions and attribute both (1, 2) and (2, 1)), violating business rule 𝐿 prohibits recievidence with schema-level are absent from text, mapping menti 1 notable contributions. Challenge 3: No one-shot guarantees. values are correctly extracted, the result violates basic data semantics. uncertain; forcing early key assignmenE (1) a Entity extraction. Figure ??(a) shows the entity table produced procity. Second, although the document implies multi-hop invest(1)assignments, The Doc2DB formalize the integrity Doc2DB proble Con!icting evidence leads to incorrect such asproblem. unit mis- We snippets, whereas constrain keyand join-consistent extraction br by an LLM extractor. Whilew.r.t. many mentions and attribute ment chain (𝐿 3 ), the extractor fails to recover the transitive relationmatch for Alpha Holdings 𝐿 2company . Moreover, entities mentioned only global veri"cation over deploymen the entire data articulate keydata system requirements for practical Challenge 3: No one-shot guarantees. E are such correctly extracted, the materialized result violatesasbasic semantics. partially, as Coyni Inc, separate tuples with ditional IE pipelines can extract frag ship (2, 3), leaving an implied edge missing. values ! are consistent entity resolution, relationship construction, and Con!icting evidence leads to incorrect assignments, such as unit missnippets, whereas integrity constraint missing attributes, introducing both incompleteness and duplication. provide no guarantees of integrity or match for AlphaLocal Holdings w.r.t. 𝐿 2 . Moreover, entities mentioned only global veri"cation thechallenges entire data constraint satisfaction (Section 2). Example 1 illustrates why Doc2DB is challenging. exTaken together,over these (2) Relationship extraction. Figure ??(b) shows the output of a DOC2DB partially, such as Coyni Inc, are materialized as separate tuples with ditional pipelines can fragm system-level support thatextract goes beyon (2)based TheonDataMosaic approach. We IE design DataMosaic (Sec traction decisions produce tables that violatemissing globalattributes, integrity con- extractor straightforward relationship surface-level textual introducing both incompleteness and duplication. provide no guarantees of integrity or cues. The extracted Hold table reveals fundamental failures. First, a two Doc2DB system organized asTaken an together, extract-verify-iterate Contributions. Wethese present DataMostc straints, resulting in hallucinated entities and relationships, missing challenges (2) Relationship extraction. Figure ??(b) shows the output of a contradictory phrasing induces a spurious mutual investment (e.g., structing relational databases do system-level support that goes from beyond An orchestrator coordinates entity and relationship ext implied links, and semantically inconsistent attributes. Such errors extractorloop. straightforward based on𝐿surface-level textual both (1, 2) and relationship (2, 1)), violating business rule 1 that prohibits recinotable contributions. cues. The extracted Hold table reveals two fundamental failures. First, with veri!cation and repair, triggering targeted re-extraction cannot be reliably addressed within extractors alone, motivating a Contributions. We present DataMosa procity. Second, although the document implies a multi-hop invest(1) The Doc2DB problem. We forma contradictory induces a spurious mutual investment relationalconsistency. databases from do violations to(e.g., ensure structing database-level ment loops. chain (𝐿phrasing to recover the transitive relationseparate veri!cation stage with explicit feedback 3 ), the extractor fails detected articulate key system requirements fo both (2, 1)), business rule 𝐿 1 that prohibits recinotable contributions. ship (1, (2,2) 3),and leaving an violating implied 2edge missing. ! consistent entity resolution, relations procity. Second, although the document implies a multi-hop invest(1) The Doc2DB problem. We formal constraint satisfaction (Section 2). Example 1 illustrates why Doc2DB is challenging. Local exment chain (𝐿 3 ), the extractor fails to recover the transitive relationarticulate key system requirements for (2) The DataMosaic approach. We d traction produce violate global integrity conship (2, 3),decisions leaving an impliedtables edge that missing. ! consistent entity resolution, relations a Doc2DB system organized as an straints, resulting in hallucinated entities and relationships, missing constraint satisfaction (Section 2). Example 1 illustrates why Doc2DB is challenging. Local exloop. An orchestrator coordinates enti implied links, and semantically inconsistent attributes. Such errors (2) The DataMosaic We de traction decisions tables thatextractors violate global con-a with veri!cation and approach. repair, triggering cannot be reliablyproduce addressed within alone,integrity motivating adetected Doc2DBviolations system organized as an e straints, resulting in hallucinated entities and relationships, missing to ensure databas separate veri!cation stage with explicit feedback loops. loop. An orchestrator coordinates entit implied links, and semantically inconsistent attributes. Such errors 2 with veri!cation and repair, triggering cannot be reliably addressed within extractors alone, motivating a detected violations to ensure database separate veri!cation stage with explicit feedback loops.

Figure 1: Document-to-Database extraction: (a) document(s); (b) target database schema; (c) Extracted Entity Tables; (d) Extracted Relationship Tables.

Constructing the database requires more than extracting local values. An LLM extractor may correctly identify many company mentions and numbers, but still produce semantically invalid tuples. For entity extraction (Fig. 1(c)), it may misinterpret units, such as treating “18,524 (in thousand)” as a raw value, or materialize weak mentions such as Coyni Inc as incomplete duplicate entities. For relationship extraction (Fig. 1(d)), surface-level cues may induce spurious mutual investments, while implicit multi-hop links may be missed. These errors show that local extraction decisions must be reconciled with schema-level semantics and integrity constraints. □ From Extraction to Database Construction. This motivates Document-to-Database (Doc2DB): given a document collection 𝐹, a target schema 𝐸, and integrity or business constraints Σ, the goal is to construct a relational database instance 𝐷 that is both faithful to the documents and valid under the schema: (𝐹, 𝐸, Σ) −−−−−−−→ 𝐷. Unlike Doc2Table, Doc2DB is a database construction problem. Entity identifiers are often implicit, relationships may be distributed across document segments, and valid outputs must satisfy global constraints rather than independent field-level decisions [11, 34, 33, 37]. Therefore, evaluating document understanding at the database level requires testing not only value extraction, but also entity alignment, relationship construction, and relational validity. Despite its practical importance, Doc2DB remains underexplored as a benchmark task. Existing information extraction benchmarks, including Rotowire [40], E2E [31], LiveSum [9], InstructIE [18], StructText [19], and DTBench [51], mainly focus on flat tables, single-table extraction, or simplified generation tasks. SQUiD [35] explores text-to-relational database generation, but mainly targets logical relational view recovery rather than realistic Doc2DB construction. As summarized in Table 1, they do not fully evaluate cross-table schema construction (i.e., normalized multi-table structures with inter-table dependencies), long-context evidence aggregation, and database-level correctness. Challenges. Building a comprehensive Doc2DB benchmark is challenging for two reasons. First, direct human annotation is difficult to scale: annotators must collect documents, define schemas, and manually construct ground-truth entity and relationship tables. Second, realistic Doc2DB instances must cover diverse reasoning requirements, including unit normalization, ambiguity resolution, multihop relation construction, temporal changes, and constraint satisfaction, while also spanning domains with different narrative styles and schema structures. Our Proposal. We present Doc2DB-Bench, a benchmark for evaluating Doc2DB capabilities beyond flat table extraction. To avoid the scalability bottleneck of manual annotation, we design a controllable DB2Doc reverse synthesis pipeline grounded in existing relational databases such as BIRD [23] and Spider [45]. Starting from schemas and database instances, the pipeline decomposes tuples into atomic evidence, assigns capability labels, serializes evidence into document plans, generates style-conditioned documents, and validates the generated documents through coverage and extraction-consistency checks.

To model realistic document complexity, Doc2DB-Bench introduces a two-pillar Doc2DB taxonomy. Intra-Table Capabilities cover cell-level extraction, normalization, inference, disambiguation, and faithfulness. Inter-Table Capabilities capture database-specific reasoning, including identity resolution, relationship linking, multi-hop composition, dynamic change, and integrity constraints. This taxonomy supports controlled benchmark construction and fine-grained diagnosis of model failures. Finally, Doc2DB-Bench evaluates outputs at the database level by measuring both entity integrity

2

Table 1: Comparison of existing benchmarks and our Doc2DB-Bench. Single-Table Extraction (ST)

MultiDomain (MD)

Cross-Table Reasoning (CR)

LongContext (LC)

DB-Level Eval (DC)

Rotowire [40] E2E [31] LiveSum [9] InstructIE [18] StructText [19] DTBench [51] SQUiD [35]

3 3 3 3 3 3 3

7 7 7 3 3 3 3

7 7 7 7 7 7 7

7 7 7 7 7 3 7

7 7 7 7 7 7 3

Doc2DB-Bench (ours)

3

3

3

3

3

Benchmark

and relational fidelity, thereby testing whether extracted facts form a schema-conformant, queryable, and relationally faithful database instance. Contributions. We summarize our contributions as follows: 1. Doc2DB Capability Taxonomy. We introduce a two-pillar taxonomy of the key capabilities required for Doc2DB construction, spanning Intra-Table value extraction and Inter-Table relational reasoning, to guide benchmark design and support fine-grained evaluation. 2. Controllable DB2Doc Synthesis Pipeline. We propose a reverse-synthesis pipeline that generates long-document Doc2DB instances from real relational schemas and database instances, reducing manual annotation cost while controlling document complexity. 3. The Doc2DB-Bench Benchmark. We construct a multi-domain benchmark for evaluating LLMs on relationally faithful Doc2DB tasks, going beyond flat, single-table extraction benchmarks. 4. Extensive Experiments. We evaluate a broad range of mainstream LLMs on Doc2DB-Bench and conduct in-depth analyses across entity- and relation-level capabilities, revealing their strengths, limitations, and open research opportunities for Doc2DB extraction.

2

Document-to-Database: The Problem

2.1

Problem Statement

Database Specification. We formalize the target database specification as S = (𝐸, Σ), where 𝐸 = (E, R) denotes the Entity-Relationship schema. E and R are the sets of entity and relationship tables, respectively, and T = E ∪ R denotes the set of tables. Each table 𝑇 ∈ T is defined by an attribute set 𝐴𝑇 and a primary key 𝐾𝑇 ⊆ 𝐴𝑇 .The constraint set Σ = Σrel ∪ Σapp consists of standard relational constraints, such as type, domain, and inclusion constraints, and application-specific rules, such as temporal conditions. Together, S specifies the target database structure and its validity conditions. Document Corpus. We consider a collection of heterogeneous documents 𝐹 = { 𝑓1 , 𝑓2 , . . . , 𝑓𝑚 } that provide the evidence for instantiating the target schema. We assume low-level processing, such as text extraction and layout analysis, has been completed. Since real-world documents may contain incomplete information, unobserved attributes naturally remain NULL. Accordingly, every extracted value should be grounded in the source documents without hallucinating unsupported information. Definition 1 (The Doc2DB Problem). Given a database specification (𝐸, Σ) and a document corpus 𝐷𝑜𝑐2𝐷𝐵

𝐹 = { 𝑓1 , . . . , 𝑓𝑚 }, the Doc2DB task is to construct a relational database instance (𝐹, 𝐸, Σ) −−−−−−−−→ 𝐷, where 𝐷 = {𝐼 𝑥 | 𝑥 ∈ 𝐸 } is the collection of instantiated entity and relationship tables. The target instance 𝐷 should satisfy (1) Constraint satisfaction: 𝐷 |= Σ; and (2) Closeness to ground truth: 𝐷 is as close as possible to the ground-truth database 𝐷 ∗ . 2.2

DB2Doc Synthesis Paradigm

To construct a benchmark for evaluating Doc2DB extraction, we adopt a reverse synthesis perspective: given a ground-truth database instance 𝐷 = 𝐼 𝑥 | 𝑥 ∈ 𝐸 with specification S = (𝐸, Σ), we synthesize a source document corpus 𝐹 from which 𝐷 ∗ can be faithfully recovered. Definition 2 (DB2Doc Synthesis). Given a ground-truth database 𝐷 ∗ , its specification S, and a capability taxonomy H , DB2Doc synthesizes a document corpus 𝐹 satisfying three properties: (1)

ND DF

CA

ME

FC NR

EF

TD

AS

RC

M ITC

Pillar-1 TA

Pillar-2

DC EM

GA TI

AC

MB

IC

SR

RI LD

US

O2M

RL

IDR

CL

ED

CR

FC (1723): Format Canonicalization US (1086): Unit Standardization EM (1810): Equivalent Mapping AC (1128): Arithmetic Calculation LD (1206): Logical Derivation TC (661): Temporal Calculation MI (161): Multi-hop Inference AS (1126): Attribute Selection DF (622): Distractor Filtering CA (118): Conflict Arbitration ND (1055): Null Detection RC (259): Redundancy Control

O2M (425): One-to-Many Allocation MB (393): Multi-Entity Binding CL (469): Conditional Linkage ED (229): Entity Disambiguation CR (250): Coreference Resolution TI (274): Transitive Inference

GA (539): Global Aggregation DC (264): Dynamic Change NR (102): Referential Integrity ME (184): Mutual Exclusion

Figure 2: A Taxonomy of Doc2DB Extraction Capabilities. completeness, every ground-truth value and relational tuple in 𝐷 ∗ is supported by evidence in 𝐹; (2) exclusiveness, 𝐹 contains no evidence supporting database facts beyond 𝐷 ∗ under S; and (3) capability awareness, the evidence for each target value/relational tuple is constructed to require specific reasoning capabilities from H for its recovery, with H formalized as a two-pillar taxonomy in Sec. 3. The completeness and exclusiveness conditions jointly ensure that 𝐷 ∗ is the unique ground-truth database recoverable from 𝐹 under S, while capability awareness enables fine-grained evaluation across taxonomy levels. Each instance may exercise any subset of capabilities from H , reflecting the naturally uneven distribution of requirements across real-world documents.

3

Doc2DB Taxonomy

Doc2DB requires models to construct both values and relations. Accordingly, we organize the required capabilities into two complementary pillars. Intra-Table Capabilities (Pillar 1) asks what values should populate an entity record, whereas Inter-Table (Pillar 2) asks how entity records should be connected. Figure 2 summarizes the resulting taxonomy and its fine-grained sub-capabilities. 3.1

Pillar1: Intra-Table Capabilities

Pillar 1 focuses on what values should populate an entity record, covering four complementary challenges. Transformative Alignment (TA). Normalizes observed values into schema-compatible representations (e.g., “18,524 in thousands” → 18,524,000). Reasoning & Inference (RI). Derives values that are not explicitly stated (e.g., computing a target value from several reported quantities). Target Discrimination (TD). Identifies the intended value among distracting or conflicting evidence (e.g., selecting the current-year rather than a historical figure). Evidence Faithfulness (EF). Avoids unsupported outputs (e.g., keeping an unreported attribute as NULL). Pillar 1 extends DTBench [51] from single-table extraction to schema-compliant entity construction in Doc2DB. 3.2

Pillar 2: Inter-Table Capabilities

Pillar 2 focuses on how entity records should be connected, covering four complementary challenges. Identity Resolution (IDR). Resolves ambiguous mentions to unique database entities (e.g., identifying “Lao Zhang” as the employee in the technical department among records with the same name). Relationship Linking (RL). Maps entity mentions and interactions into normalized relational tuples (e.g., expanding “Alice manages X, Y, and Z” into three separate relationship rows). Structural Reasoning (SR). Infers relations from distributed or compositional evidence (e.g., “Alice leads the Mobile Team” and “the Mobile Team manages Project X” imply (Alice, Project X)). Integrity Constraint (IC). Suppresses relations that violate negative evidence or explicit constraints (e.g., not linking Bob to Project X when the document states that all managers except Bob are assigned to it). Detailed sub-capabilities and examples are provided in Appendix B.

4

Document-to-Database (Db2Doc) Synthesis

Given a ground-truth relational database instance 𝐷, its specification 𝑆 = (𝐸, Σ), and the capability taxonomy H , our goal is to synthesize a document corpus 𝐹 from which 𝐷 can be faithfully recovered. As shown in Fig. 3, our DB2Doc pipeline consists of five stages: capability assignment, evidence decomposition, strategic serialization, iterative document generation, and dual validation.

Reject & Rewrite Labeling Agent

Refiner Agent

c1 c2 c3 c4 c5

C1,3 RI-AR, TA-UT

SR

Serializer Agent r1

C1,3

Atomic Evidence

Writer Agent Doc Length

r2

Config

r3 r4 r5

Capability Matrix BIRD/Spider/

WikiSQL

Horizontal

e1,3

Verifier

Doc Style Noise Level

Coverage Verification

......

inject ...

Vertical

LLMs

Stochastic Vertical

Sliding Window

Ground-Truth Table

Step 1: Capability

Assignment

Evidence Pool

Serialized Queue

Step 2: Evidence Decomposition

Step 3: Strategic Serialization

Step 4 : Iterative

Generation

Extraction Consistency

pass

Synthesized Provenance

Document Log

Step 5 : Dual

Validation

Figure 3: Overview of the benchmark construction framework. The rationale behind this design is to separate what should be tested from how it is expressed in documents. Capability assignment determines the target reasoning requirements; evidence decomposition grounds each target value or tuple in atomic evidence; serialization controls the document structure; generation turns the evidence into realistic long-form text; and validation checks that the final document remains complete, faithful, and recoverable. This staged design makes the synthesis process controllable, reproducible, and suitable for fine-grained Doc2DB evaluation. Step 1: Capability Assignment. † A Labeling Agent scans each cell 𝑣 𝑖 𝑗 ∈ 𝐷 and assigns a label set L𝑖 𝑗 ⊆ H to produce an annotated matrix M𝐷 . The agent employs an LLM as a semantic router that jointly conditions on the cell value 𝑣𝑖 𝑗, its attribute context from S, and the enclosing relational tuple 𝑟, reasoning over both intra-table semantics and inter-table relational structure to determine L𝑖 𝑗. Multi-label assignment captures compositional requirements, e.g., a cell demanding summation followed by currency conversion is labeled L𝑖 𝑗 = RI-AR, TA-UT, as illustrated in Sec. 3. Step 2: Joint Evidence Decomposition. üA Refiner Agent processes each annotated cell 𝐶𝑖 𝑗 and decomposes it into an atomic evidence set 𝐸 𝑖 𝑗 = 𝑒 1 , 𝑒 2 , . . . , 𝑒 𝑘 , forming the global evidence pool P. Critically, the agent performs joint execution over two orthogonal constraint dimensions simultaneously: the attribute-level label (Pillar 1, e.g., TA-EM for enumeration mapping) and the row-level label (Pillar 2, e.g., SR for structural reasoning). To ensure evidence quality, we apply checklist-based verification along four dimensions: (i) value/relation correctness, ensuring factual consistency with target values and relations; (ii) label faithfulness, verifying that the assigned capability is properly instantiated; (iii) schema linkage, checking consistency with valid schema elements and cross-table references; and (iv) data integrity and null faithfulness, preventing unsupported content and preserving missing values. Failed cases receive diagnostic feedback and are iteratively revised until verification succeeds or a retry limit is reached. Detailed criteria are provided in Appendix E.2. Step 3: Strategic Serialization. ˚ A Serializer Agent linearizes the verified evidence pool P into coherent blocks, controlling the document’s macro-structure and evidence organization. Each evidence item 𝑒 ∈ P is indexed by its entity 𝑖 and attribute 𝑗, and serialized according to 𝑉 (𝑒) = (𝜋1 , 𝜋2 , 𝜖), where (𝜋1 , 𝜋2 ) denotes a mode-specific ordering of the entity and attribute indices (𝑖, 𝑗), and 𝜖 ∼ 𝑈 (0, 1) serves as a stochastic tie-breaker. We consider three serialization strategies: Horizontal (𝑉 = (𝑖, 𝑗, 𝜖)), which groups evidence by entity; Vertical (𝑉 = ( 𝑗, 𝑖, 𝜖)), which groups evidence by attribute across entities; and Stochastic Vertical (𝑉 = (𝜋( 𝑗), 𝑖, 𝜖)), which follows the vertical scheme while randomly permuting the attribute order. Together, these strategies introduce controllable variation in document structure while maintaining coherent evidence organization. Step 4: Iterative Document Generation. œ A Writer Agent generates the final document block by block following the serialized plan. A context-aware sliding window conditions each block on the current evidence 𝐸 𝑏𝑙𝑜𝑐𝑘 and a bounded history of previously generated text 𝐻 𝑝𝑟 𝑒𝑣 , promoting fluent transitions and consistent cross-block coreference. Furthermore, to better emulate the complexity of real-world documents, generation is controlled by a plug-and-play configuration module. It supports configurable parameters such as

Table 2: Comprehensive statistics of the Doc2DB-Bench benchmark. The # denotes the total count, and Avg. Len. indicates the average document length per sample in tokens. Document

Domain

Schema

Table Scale (Total)

# Docs

Avg. Len.

# DBs

# Ent.

# Rel.

# Rows

# Cols

# Cells

Education Finance Medical University Sports Transportation Others

13 59 11 31 27 17 43

40,424 40,073 43,930 33,913 50,991 53,091 46,566

3 10 3 6 5 5 10

12 23 9 17 20 12 24

12 39 12 21 16 6 26

830 1,918 343 860 1,085 718 1,587

96 353 199 209 177 128 313

3,451 10,454 3,522 4,339 5,772 4,678 9,719

Total/Avg.

203

43,326

42

117

132

7,341

1,475

41,935

Document_Length, Noise_Level (∈ [0, 1]), and Document_Style. The noise level controls the frequency of natural distractors, including irrelevant background content, formatting artifacts, and header/footer interference, while the style parameter follows domain-specific reference documents (e.g., formal legal writing or concise clinical shorthand). Together, these controls increase document realism and diversity without violating the factual constraints in the evidence pool P. Step 5: Dual Validation. To guarantee that the synthesized document 𝐷 𝑠𝑦𝑛 satisfies the rigorous constraints of the Doc2DB paradigm, ϻ a Verifier Agent conducts a final dual-validation pass. This phase operates as an inverse-consistency check: • Coverage Verification: Ensures that no required atomic evidence from P is omitted or semantically distorted, utilizing explicit boundary tags (e.g., <frag_𝑖>…</frag_𝑖>) injected during generation to deterministically trace textual spans back to their source atoms. • Extraction Consistency: An evaluation mechanism is deployed to re-extract the database instance 𝐷 ′𝑠𝑦𝑛 from 𝐷 𝑠𝑦𝑛 . We rigorously verify the structural isomorphism and value equivalence between the re-extracted 𝐷 ′𝑠𝑦𝑛 and the ground-truth database 𝐷. If 𝐷 syn fails either validation criterion due to hallucinated relations, missing evidence, or referential inconsistencies, it is routed back to Step 4 for targeted rewriting with error feedback. This closed-loop validation minimizes unrecoverable or contradictory information in the final benchmark corpus. Doc2DB-Bench Statistics. We synthesize documents from databases sourced from BIRD [23] and Spider [45], using them as ground truth. Specifically, we curate 42 high-quality databases across seven domains, including education and medical. Details of database collection and processing are provided in Appendix A. All agents use Gemini-2.5-Pro [7] as the backbone model, selected based on preliminary experiments balancing quality and cost. All synthesized cases are further verified by seven computer science PhD candidates, each specializing in a corresponding domain. As summarized in Table 2, Doc2DB-Bench contains 42 synthesized (Document, Specification, Database) samples, covering 117 entity tables, 132 relationship tables, 7,341 rows, and 41,935 cells across seven domains. The benchmark targets schema understanding, long-context reasoning, and cross-table dependency modeling, with fine-grained capability annotations for 11,205 cells (>25%) and 3,129 rows (>40%). To guide realistic synthesis, we collect reference documents from DocBench [52], MMLongBench [29], MMDocRAG [10], and web sources, spanning financial and legal reports to research abstracts. Fig. 2 shows the capability distribution; unannotated cells and rows correspond to directly extractable cases requiring verbatim recovery.

5

Experiments

In this section, we systematically evaluate the performance of various LLMs on Doc2DB-Bench for the Doc2DB extraction task. Specifically, we seek to answer the following research questions: RQ1: How well do different LLMs perform on Doc2DB extraction at the entity and relation levels? RQ2: Beyond overall performance, what specific capabilities and vulnerabilities do these models exhibit across the fine-grained dimensions defined in our taxonomy? RQ3: How realistic and authentic are the documents generated by our automated pipeline?

Table 3: Comparative performance of LLMs on Doc2DB-Bench. Green highlights the best. Model

Size

Entity-Level P.

Relation-Level

R.

F1

P.

R.

Overall Performance F1

P.

R.

F1

LS

Open-source Models Qwen2.5-14b-Ins

14b

42.87

60.17

44.07

27.45

43.54

28.92

29.40

50.36

32.97

39.39

LLaMA-3.1-70b-Ins Qwen2.5-72b-Ins

70b 72b

14.96 41.62

58.89 66.57

17.25 45.03

20.29 28.48

38.79 54.63

16.97 31.98

12.12 30.96

47.46 59.53

14.29 36.07

22.76 39.53

GPT-4o [16]

-

72.39

70.92

70.30

62.94

46.78

50.68

65.66

55.99

59.05

47.52

GPT-5.4 Gemini-2.5-flash [7]

-

79.93 75.89

81.12 75.48

80.09 74.21

80.09 73.60

69.63 57.35

72.33 59.50

78.47 71.33

73.93 64.58

75.25 65.99

69.02 53.74

Gemini-2.5-pro [7]

-

76.80

82.24

78.13

67.98

67.01

65.46

69.83

73.80

70.81

57.19

Claude-opus-4-6 Qwen3-max

-

87.19 76.51

83.70 77.00

84.93 75.77

66.71 57.01

62.16 59.74

63.17 55.96

77.33 63.48

71.57 66.42

73.60 63.44

61.95 52.48

Deepseek-V4-flash

-

76.65

66.06

67.30

67.67

43.23

48.93

69.03

52.14

57.06

49.05

Proprietary Models

Specialized IE Systems LangExtract [12]

-

29.72

40.50

33.49

33.34

19.08

22.00

36.06

25.20

28.72

23.55

DocETL [36]

-

78.20

76.85

74.91

46.75

41.26

41.51

65.27

60.97

60.82

49.87

LangChain [14] LlamaExtract [28]

-

60.77 52.28

83.09 68.95

69.09 56.58

33.50 47.41

42.47 48.73

36.33 47.39

47.28 45.35

64.38 56.49

53.83 49.10

44.14 40.26

5.1

Setup

Baselines. We benchmark a broad range of state-of-the-art LLMs, categorized into three groups: (1) Open-weight models, including Qwen2.5-14B/72B-Instruct [44] and Llama-3.1-70B-Instruct; (2) Proprietary APIs, including GPT-4o [16], GPT-5.4, Gemini-2.5-Flash/Pro [7], Claude-opus-4-6 [2], Qwen3-Max, and DeepSeek-V4-Flash [42]; and (3) Specialized IE systems, including LlamaExtract [28], a commercial extraction service with iterative refinement and citations; LangExtract [12], which uses chunking and few-shot prompting for long-document extraction; LangChain [14], a general framework with schema-constrained function calling for structured outputs; and DocETL [36], a declarative LLM-based data processing framework with modular extraction pipelines. Evaluation Details. For fair and reproducible comparison, we use identical prompts and greedy decoding with temperature 0 across all experiments. All systems use GPT-5.4 as the backbone model, except LlamaExtract, which uses its proprietary model. To isolate relational reasoning from upstream entity extraction errors, we additionally introduce an oracle entity setting, where the model M receives ground-truth entity tables 𝐸 ∗ and predicts only relation tables: 𝑇rel = M (𝐷, 𝑆, 𝐸 ∗ ). Metrics. To evaluate extraction at a granular level, we employ the following metrics: • Cell-level Accuracy. We evaluate fine-grained attribute extraction using cell-level Precision, Recall, and F1 . Rather than relying on row-level matching, which is sensitive to tuple ordering and may cause cascading alignment errors, we employ a Global Maximum Weight Matching algorithm to establish one-to-one tuple alignment between predicted (𝑇) and ground-truth (𝑇 ∗ ) tables (details in Appendix C.1). Let 𝐶match denote the number of cells satisfying our matching criteria (e.g.,, exact numeric equality or ≥90% string similarity). The metrics are defined as: 𝑃=

|𝐶match | , 𝐶total (𝑇)

𝑅=

|𝐶match | , 𝐶total (𝑇 ∗ )

𝐹1 =

2𝑃𝑅 , 𝑃+𝑅

where 𝐶total (·) denotes the number of non-empty cells in the corresponding table. • Semantic Quality. We employ GPT-4o as an evaluator to assess semantic equivalence beyond exact cell matching. It assigns a score from 0 to 100 based on factual correctness, schema conformity, and robustness to paraphrasing, capturing semantically equivalent but structurally varied predictions. The evaluation prompt is provided in Appendix C.2, with further results in Appendix D. 5.2

Experimental Results and Analysis

Overall Performance. As shown in Table 3, proprietary models consistently outperform opensource counterparts across all metrics. In particular, GPT-5.4 achieves the best overall F1 score

Table 4: The Comparison of Capabilities Across Different Models for Different Categories. Cell colors represent performance levels (Blue: High; Red: Low). Capability

Sub-capabilities

Qwen2.5-

LLaMA-3.1-

14B-Ins

70B-Ins

GPT-4o

DeepSeek-

Qwen3-Max

V4-Flash

Claude-

Gemini-

Opus-4-6

2.5-Pro

GPT-5.4

Identity

Entity Disambiguation

51.58

59.92

69.24

55.25

64.57

72.96

80.03

79.49

Resolution

Coreference Resolution

37.84

50.13

61.14

62.10

56.76

67.61

77.71

82.18

One-to-Many Allocation

49.06

51.90

65.63

62.27

64.77

70.38

78.18

81.83

Multi-Entity Binding

47.65

53.85

60.18

53.83

58.57

64.82

67.26

82.97

Conditional Linkage

52.02

57.47

69.10

62.65

61.80

69.64

75.81

79.16

Transitive Inference

40.80

57.83

65.51

57.35

53.78

59.48

70.82

78.47

Global Aggregation

49.23

48.30

71.59

59.83

62.15

64.25

73.03

82.20

Dynamic Change

36.51

51.83

58.11

46.93

58.36

66.26

78.69

79.27

Integrity

Referential Integrity

-

-

-

-

-

-

-

-

Constraint

Mutual Exclusion

36.41

56.94

60.18

58.45

54.61

61.96

70.39

74.46

Relationship Linking

Structural Reasoning

80

Entity-Level w/o label w/ label

75

70

65

65

60

60

55

55

Figure 4: Radar chart of different models across various capabilities.

GPT-5.4

Qwen3-max

w/o label w/ label

75

70

50

Relation-Level

80

50

GPT-5.4

Qwen3-max Gemini-2.5-pro

Figure 5: Impact of capability annotations on extraction performance.

(75.25) and the highest llm score (69.02), indicating strong capability in structured extraction. Claude-opus-4-6 and Gemini-2.5-pro also deliver competitive performance. Notably, GPT-4o shows only moderate results (Overall F1 59.05), falling noticeably short of reasoning-capable models despite its scale. In contrast, open-source models such as Qwen2.5-14B-Ins exhibit significantly lower overall F1 (32.97), highlighting the persistent performance gap under complex extraction settings. Finding 1. Models with stronger reasoning capabilities consistently achieve better performance on Doc2DB extraction, where structural reasoning proves critical for schema-compliant output. Entity-level performance. Entity-level results reveal a clear advantage of high-capacity proprietary models. Claude-opus-4-6 achieves the highest entity-level F1 (84.95), followed by GPT-5.4 (80.09) and Gemini-2.5-pro (78.13). These models maintain a good balance between precision and recall, indicating robust span detection and boundary alignment. Compared to them, Qwen2.5-14B shows notably lower precision (42.87) despite relatively higher recall (60.17), suggesting that open-source models tend to over-generate entity spans, leading to reduced accuracy. Relationship level performance. Relation extraction is evaluated under an oracle entity setting: given the relational schema, the source document, and the ground-truth entity tables, the model generates only the relation tables. This setting isolates relational reasoning ability from upstream entity extraction errors. As shown in Table 3, relation-level extraction remains substantially more challenging than entity extraction for most models. A key reason is that relation prediction requires not only identifying the correct records, but also globally aligning related entities and tuples under a structured schema, which is more error-prone than localized entity extraction. GPT-5.4 achieves the best relation-level F1 (72.33), outperforming GPT-4o (50.68), Gemini-2.5-pro (65.46), and Claude-opus4-6 (63.13), indicating stronger capability in global alignment and structured relational reasoning. Finding 2. Relation-level extraction is a major bottleneck in Doc2DB extraction. Compared with entity extraction, it is substantially more challenging because it requires accurate global alignment across entities and tuples, leading to consistently lower performance.

Table 5: Quality and Authenticity evaluation of Doc2DB-Bench documents using (a) rubric-based LLM and human assessment and (b) commercial AI detectors. Judger LLM Human

5.3

Lexical Logical Textual Genre Average Richness Consistency Coherence Fidelity 4.46 4.17

4.28 3.97

3.84 3.98

4.94 4.00

4.38 4.03

Method ZeroGPT FastGPT [4]

Doc2DB-Bench Reference 35.20 37.50

23.12 35.38

Capability-Level Performance

As revealed in Section 5.2, relation extraction is the primary bottleneck in Doc2DB extraction, yet overall metrics do not reveal which specific inter-table capabilities remain challenging for LLMs. We therefore conduct a fine-grained evaluation over Pillar 2 capabilities in our taxonomy. For each capability, we compute cell-level Precision, Recall, and F1 on the corresponding annotated relation tables using the same matching criteria, enabling fine-grained analysis of inter-table reasoning abilities such as entity linking, relation alignment, and multi-hop dependency recovery. Capability assignment introduces difficulty. As shown in Fig. 5, models perform consistently worse on documents with explicit capability labels than on those without. The drop is substantial at the entity level, with GPT-5.4 decreasing from 74.15 to 56.75 and Qwen3-max from 66.70 to 52.37. At the relation level, the decline is smaller but consistent across models (GPT-5.4: 67.77 to 58.69; Qwen3-max: 56.63 to 51.23; Gemini-2.5-pro: 66.49 to 60.26). These results confirm that capability-annotated documents are genuinely harder and validate the effectiveness of our taxonomy. Detailed Error Analysis. Fig. 4 shows that GPT-5.4 performs best across all capabilities, while Table 4 reports fine-grained results across four inter-table capabilities, revealing three key findings: (i) Implicit evidence amplifies model disparities. Surface-level tasks rely on explicit cues, whereas coreference resolution and multi-entity binding require aggregating scattered implicit evidence, significantly widening the gap between strong and weak models. (ii) Structural reasoning remains a major bottleneck. Even SOTA models plateau at around 80% on Dynamic Change and Transitive Inference, while smaller models fall below 40%, indicating limited multi-step relational reasoning ability. (iii) Integrity Constraint (IC) verification is consistently weak. Models often hallucinate dependencies in Referential Integrity tests and violate exclusivity rules in Mutual Exclusion tasks, highlighting the lack of constraint-aware generation and potential risks to database reliability. Finding 3.. Implicit reasoning over distributed and compositional evidence increases difficulty and widens model gaps, with structural reasoning and integrity constraints as the primary failure modes. 5.4

Authenticity Verification

To evaluate document quality and realism, we conduct a rubric-based assessment using both an LLM judge and human annotators. Documents are evaluated along four dimensions: lexical richness, logical consistency, textual coherence, and genre fidelity, using a five-point scale. We use DeepSeekV3.2 as the automatic evaluator and recruit five graduate students document intelligence research experience to independently assess 50 randomly sampled documents using the same rubrics. As shown in Table 5(a), the LLM and human evaluations consistently demonstrate the high linguistic quality and domain authenticity. Furthermore, commercial AI detectors (ZeroGPT and FastGPT [4]) show that Doc2DB-Bench closely matches real-world distributions in AI-generation scores (Table 5(b)). Notably, the FastGPT score (37.50%) is nearly identical to the real-data baseline (35.38%). Overall, these results demonstrate that Doc2DB-Bench preserves the lexical diversity and stylistic nuances of human-written reports, achieving high fidelity. More details are provided in Appendix F.

6

Related Work

Structured Information Extraction. Traditional IE decomposes the problem into sub-tasks such as named entity recognition and relation extraction, evolving from rule-based methods [15, 21] to pretrained language model-based approaches [24, 41, 49]. Such pipelines are difficult to customize for schema-driven extraction, where target attributes may fall outside existing ontologies [18]. Recent work leverages LLMs for structured extraction: some advanced prompting methods [9, 1, 39] decompose complex extraction tasks into intermediate steps; StructSum [17] generates structured summaries as tables and mind maps; InstructIE [18], Lotus [32], and PZ [27] extract specified attributes

through natural-language queries; Doctopus [5] combines LLMs with traditional IE tools to reduce cost. However, existing work focuses on Doc2Table extraction over single flat tables [39, 9, 18, 26], while the Doc2DB setting, which requires constructing multi-table relational databases with intertable dependencies and integrity constraints from documents, remains largely unexplored. Existing Benchmarks. Prior benchmarks largely repurpose table-to-text datasets such as Rotowire [40] and Wiki40B [13], which support only trivial extraction reducible to direct text replication. LiveSum [9] covers limited domains, while synthetic benchmarks InstructIE [18] and StructText [19] suffer from simplified schemata and trivial text–tuple alignments. Meanwhile, SemBench [20] and UDA-Bench [38] target unstructured document analysis but lack fine-grained capability evaluation. More recent DTBench [51] introduces capability-aware evaluation for document-to-table extraction, but remains limited to flat-table extraction without cross-table relational construction. SQUiD [35] further explores text-to-relational database generation with multi-table schema synthesis, yet mainly evaluates logical relational view recovery from simple descriptions rather than realistic Doc2DB task requiring cross-table reasoning and database-level consistency. These limitations motivate Doc2DBBench, the first benchmark for systematic LLM evaluation on document-to-database extraction.

7

Conclusion

We introduce Doc2DB-Bench, a novel benchmark advancing information extraction from isolated flat tables to relationally faithful document-to-database construction. To overcome manual annotation bottlenecks, we propose a controllable reverse-synthesis pipeline grounded in a comprehensive taxonomy of intra-table and inter-table capabilities. Extensive evaluations reveal that while state-of-the-art LLMs excel at localized entity extraction, they fundamentally struggle with global relational alignment, multi-hop structural reasoning, and adherence to database integrity constraints. Doc2DB-Bench provides a rigorous and realistic testbed for advancing reliable, schema-compliant, and auditable LLM-based data systems, with further dataset expansion underway.

Limitations While Doc2DB-Bench provides broad coverage across diverse schemas and domains, several aspects remain open for further exploration. The current DB2Doc synthesis pipeline relies on LLM-based generation, making human verification important for ensuring the quality and realism of synthesized documents. Beyond synthesis quality, we also plan to expand the benchmark by incorporating more domain-specific databases and broader real-world sources, strengthening its coverage and diversity.

References [1] N. Ahuja, F. Bardoliya, C. Baral, and V. Gupta. Map&make: Schema guided text to table generation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 30249–30262, 2025. [2] Anthropic. Claude opus 4.6 system card. https://www-cdn.anthropic.com/ 6a5fa276ac68b9aeb0c8b6af5fa36326e0e166dd/Claude%20Opus%204.6%20System% 20Card.pdf, 2026. [3] H. Babaei Giglou, J. D’Souza, and S. Auer. Llms4synthesis: Leveraging large language models for scientific synthesis. In Proceedings of the 24th ACM/IEEE Joint Conference on Digital Libraries, pages 1–12, 2024. [4] G. Bao, Y. Zhao, Z. Teng, L. Yang, and Y. Zhang. Fast-detectgpt: Efficient zero-shot detection of machine-generated text via conditional probability curvature. arXiv preprint arXiv:2310.05130, 2023. [5] C. Chai, J. Li, Y. Deng, Y. Zhong, Y. Yuan, G. Wang, and L. Cao. Doctopus: Budget-aware structural table extraction from unstructured documents. Proceedings of the VLDB Endowment, 18(11):3695–3707, 2025. [6] Z. Chen, Z. Gu, L. Cao, J. Fan, S. Madden, and N. Tang. Symphony: Towards natural language query answering over multi-modal data lakes. In CIDR, pages 1–7, 2023.

[7] G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025. [8] C. Deng, J. Yuan, P. Bu, P. Wang, Z.-Z. Li, J. Xu, X.-H. Li, Y. Gao, J. Song, B. Zheng, et al. Longdocurl: a comprehensive multimodal long document benchmark integrating understanding, reasoning, and locating. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1135–1159, 2025. [9] Z. Deng, C. Chan, W. Wang, Y. Sun, W. Fan, T. Zheng, Y. Yim, and Y. Song. Text-tupletable: Towards information integration in text-to-table generation via global tuple extraction. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9300–9322. Association for Computational Linguistics, 2024. [10] K. Dong, Y. Chang, S. Huang, Y. Wang, R. Tang, and Y. Liu. Benchmarking retrievalaugmented multimodal generation for document question answering. arXiv preprint arXiv:2505.16470, 2025. [11] X. Dong, E. Gabrilovich, G. Heitz, W. Horn, N. Lao, K. Murphy, T. Strohmann, S. Sun, and W. Zhang. Knowledge vault: A web-scale approach to probabilistic knowledge fusion. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 601–610, 2014. [12] Google. Introducing langextract. https://langextract.net, 2025. [13] M. Guo, Z. Dai, D. Vrandečić, and R. Al-Rfou. Wiki-40b: Multilingual language model dataset. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 2440– 2452, 2020. [14] Harrison Chase. Langchain. https://github.com/langchain-ai/langchain, 2022. [15] J. Hobbs. Fastus: A finite-state processor for information extraction from real-world text. In … JOINT CONFERENCE ON…, 1993. [16] A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. [17] P. Jain, A. Marzoca, and F. Piccinno. Structsum generation for faster text comprehension. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7876–7896, 2024. [18] Y. Jiao, M. Zhong, S. Li, R. Zhao, S. Ouyang, H. Ji, and J. Han. Instruct and extract: Instruction tuning for on-demand information extraction. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10030–10051. Association for Computational Linguistics, 2023. [19] S. Kashyap, S. Shirai, N. Mihindukulasooriya, and H. Samulowitz. Structtext: A synthetic table-to-text approach for benchmark generation with multi-dimensional evaluation. CoRR, abs/2507.21340, 2025. [20] J. Lao, A. Zimmerer, O. Ovcharenko, T. Cong, M. Russo, G. Vitagliano, M. Cochez, F. Özcan, G. Gupta, T. Hottelier, et al. Sembench: A benchmark for semantic query processing engines. arXiv preprint arXiv:2511.01716, 2025. [21] T. Lee, Z. Wang, H. Wang, and S.-w. Hwang. Attribute extraction and scoring: A probabilistic approach. In 2013 IEEE 29th International Conference on Data Engineering (ICDE), pages 194–205. IEEE, 2013. [22] B. Li, Z. Liang, Y. Xie, X. Lin, T. Luo, X. Liu, Y. Zhu, Z. Peng, Y. Li, Z. Zhang, et al. Dataspace: Benchmarking data agents for verifiable analytics over heterogeneous workspaces. arXiv preprint arXiv:2608.03451, 2026.

[23] J. Li, B. Hui, G. Qu, J. Yang, B. Li, B. Li, B. Wang, B. Qin, R. Geng, N. Huo, et al. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. Advances in Neural Information Processing Systems, 36:42330–42357, 2023. [24] Y. Li, L. Song, and C. Zhang. Sparse conditional hidden markov model for weakly supervised named entity recognition. In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining, pages 978–988, 2022. [25] Z. Li, X. Chen, H. Yu, H. Lin, Y. Lu, Q. Tang, F. Huang, X. Han, L. Sun, and Y. Li. Structrag: Boosting knowledge intensive reasoning of llms via inference-time hybrid information structurization. In International Conference on Learning Representations, volume 2025, pages 36107–36124, 2025. [26] Z. Liang, X. Lin, Z. Zhang, Y. Luo, H. Wang, and N. Tang. Long-document qa with chain-ofstructured-thought and fine-tuned slms. arXiv preprint arXiv:2603.29232, 2026. [27] C. Liu, M. Russo, M. Cafarella, L. Cao, P. B. Chen, Z. Chen, M. Franklin, T. Kraska, S. Madden, R. Shahout, et al. Palimpzest: Optimizing ai-powered analytics with declarative query processing. In Proceedings of the Conference on Innovative Database Research (CIDR), page 2, 2025. [28] LlamaIndex. Unstructured data extraction: Turn documents into insights. https://www. llamaindex.ai/blog/unstructured-data-extraction, 2024. [29] Y. Ma, Y. Zang, L. Chen, M. Chen, Y. Jiao, X. Li, X. Lu, Z. Liu, Y. Ma, X. Dong, et al. Mmlongbench-doc: Benchmarking long-context document understanding with visualizations. Advances in Neural Information Processing Systems, 37:95963–96010, 2024. [30] H. Muscolino, A. Machado, D. Vesset, and J. Rydning. What every executive needs to know about unstructured data. Technical report, IDC Corp., sponsored by Box Inc., Aug. 2023. https://www.storagenewsletter.com/2023/09/07/ what-every-executive-needs-to-know-about-unstructured-data/. [31] J. Novikova, O. Dusek, and V. Rieser. The E2E dataset: New challenges for end-to-end generation. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 201–206. Association for Computational Linguistics, 2017. [32] L. Patel, S. Jha, M. Pan, H. Gupta, P. Asawa, C. Guestrin, and M. Zaharia. Semantic operators and their optimization: Enabling llm-based data processing with accuracy guarantees in lotus. Proceedings of the VLDB Endowment, 18(11):4171–4184, 2025. [33] N. Peng, H. Poon, C. Quirk, K. Toutanova, and W.-t. Yih. Cross-sentence n-ary relation extraction with graph lstms. Transactions of the Association for Computational Linguistics, 5:101– 115, 2017. [34] C. D. Sa, C. Zhang, J. Shin, and C. Ré. Deepdive: Declarative knowledge base construction. SIGMOD Record, 45(1):60–67, 2016. [35] M. Sadia, Z. Yang, Y. Xiao, A. Chen, and A. R. Chowdhury. Squid: Synthesizing relational databases from unstructured text. In C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng, editors, Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, EMNLP 2025, Suzhou, China, November 4-9, 2025, pages 31987–32012. Association for Computational Linguistics, 2025. [36] S. Shankar, T. Chambers, T. Shah, A. G. Parameswaran, and E. Wu. Docetl: Agentic query rewriting and evaluation for complex document processing. arXiv preprint arXiv:2410.12189, 2024. [37] J. Shin, S. Wu, F. Wang, C. De Sa, C. Zhang, and C. Ré. Incremental knowledge base construction using deepdive. In Proceedings of the VLDB Endowment International Conference on Very Large Data Bases, volume 8, page 1310, 2015. [38] V. Subramaniyaswamy, V. Vijayakumar, R. Logesh, and V. Indragandhi. Unstructured data analysis on big data using map reduce. Procedia Computer Science, 50:456–465, 2015.

[39] X. Tang, Y. Zong, J. Phang, Y. Zhao, W. Zhou, A. Cohan, and M. Gerstein. Struc-bench: Are large language models good at generating complex structured tabular data? In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2024. [40] S. Wiseman, S. M. Shieber, and A. M. Rush. Challenges in data-to-document generation. arXiv preprint arXiv:1707.08052, 2017. [41] F. Wu, R. Hoffmann, and D. S. Weld. Information extraction from wikipedia: Moving down the long tail. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 731–739, 2008. [42] A. Xu, B. Lin, B. Xue, B. Wang, B. Xu, B. Wu, B. Zhang, C. Lin, C. Dong, C. Ling, et al. Deepseek-v4: Towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348, 2026. [43] D. Xu, W. Chen, W. Peng, et al. Large language models for generative information extraction: A survey. Frontiers of Computer Science, 18, 2024. https://journal.hep.com.cn/fcs/ EN/10.1007/s11704-024-40555-y. [44] A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Xia, X. Ren, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Wan, Y. Liu, Z. Cui, Z. Zhang, and Z. Qiu. Qwen2.5 technical report. CoRR, abs/2412.15115, 2024. [45] T. Yu, R. Zhang, K. Yang, M. Yasunaga, D. Wang, Z. Li, J. Ma, I. Li, Q. Yao, S. Roman, et al. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 3911–3921, 2018. [46] T. Zeng, Y. Hong, Z. Ding, D. Liu, Y. Mei, Y. Su, Y. Wang, X. Zhang, J. Zhu, J. Zhu, et al. Qwenpaw-data: Bridging facts, methodology, and execution for autonomous enterprise data analytics. arXiv preprint arXiv:2607.11019, 2026. [47] Z. Zhang, Z. Liang, J. Chen, H. Wang, and N. Tang. Document-to-database: Extraction meets relational semantics. volume 19, pages 2522–2535. VLDB Endowment, 2026. [48] Z. Zhang, Z. Liang, Y. Wu, T. Lin, Y. Luo, and N. Tang. Datamosaic: Explainable and verifiable multi-modal data analytics through extract-reason-verify. arXiv e-prints, pages arXiv–2504, 2025. [49] G. Zheng, S. Mukherjee, X. L. Dong, and F. Li. Opentag: Open attribute value extraction from product profiles. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1049–1058, 2018. [50] L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595–46623, 2023. [51] ZJU-DAILY. DTBench: A synthetic benchmark for document-to-table extraction, 2026. https://arxiv.org/abs/2602.13812. [52] A. Zou, W. Yu, H. Zhang, K. Ma, D. Cai, Z. Zhang, H. Zhao, and D. Yu. Docbench: A benchmark for evaluating llm-based document reading systems. In Proceedings of the 4th International Workshop on Knowledge-Augmented Methods for Natural Language Processing, pages 359–373, 2025.

A

Dataset Collection and Processing

To establish a relationally faithful ground truth for Doc2DB-Bench, we curated a collection of complex relational databases from two predominant text-to-SQL benchmarks: BIRD [23] and Spider [45]. The collection process followed a multi-stage pipeline designed to maximize domain diversity and structural complexity.

Count

Count

8 6 4 2 0

0 50 100 150 200 250 300 350 400 >450

# Rows

9 8 7 6 5 4 3 2 1 0

12 10 8

Count

10

6 4 2

10 15 20 25 30 35 40 45 50 55 >60

# Columns

0

0 250 500 750 1000 1250 1500 1750>2000

# Cells

Figure 6: Distribution of database scale in Doc2DB-Bench in terms of rows, columns, and cells. A.1

Database Selection and Sourcing

We first screen the database pools from BIRD and Spider, selecting databases that are suitable for document-to-database evaluation. Specifically, we prioritize databases with at least three interconnected tables, non-trivial schemas involving multiple entity types, relationship tables, and foreign-key dependencies, as well as diverse cross-table dependencies. To ensure broad coverage, the final collection spans seven domains, including education, finance, healthcare, university, sports, transportation, and others. Unlike conventional benchmarks that focus on flat-table extraction, our selection favors databases with rich entity-relationship (ER) structures and explicit relational dependencies, enabling the evaluation of cross-table reasoning and database construction capabilities. A.2

Data Cleansing and Schema Refinement

The raw databases underwent several refinement steps to ensure they were suitable for document synthesis: • Schema Standardization: We normalize attribute names, unify schema representations, and ensure that primary and foreign keys are explicitly defined. This step reduces schema ambiguity and facilitates reliable DB-to-document synthesis. • Tuple Sampling and Instance Construction: Since using all database tuples may produce excessively long documents and dilute evidence density, we sample representative tuples while preserving entity coverage, inter-table relationships, and referential integrity. The resulting instances remain complete relational databases rather than isolated table fragments, yielding 117 entity tables and 132 relationship tables with 7,341 rows and 41,935 cells. Fig. 6 further illustrates the distribution of database scales in terms of rows, columns, and cells. • Integrity Verification: We perform SQL-based sanity checks on the sampled databases to ensure that all instances satisfy relational constraints, including primary-key uniqueness, foreign-key referential integrity, and attribute-level validity. We further verify schema consistency, table connectivity, and tuple completeness to ensure that sampled instances preserve the original relational structures without introducing broken dependencies or isolated fragments. Only verified database instances are retained for subsequent document synthesis. A.3

Domain-Specific Style Grounding

Although databases from BIRD [23] and Spider [45] provide diverse domain schemas and structured facts, they are not designed to capture realistic document styles and writing conventions. To improve document realism, we augment the structured data with domain-specific narrative styles by collecting reference documents from LongDocURL [8], DocBench [52], MMLongBench [29], and MMDocRAG [10]. These references cover diverse document formats, ranging from academic papers and financial reports to posters and restaurant menus, serving as stylistic anchors for the synthesis pipeline. They enable the generation of authentic domain-specific documents while strictly preserving the factual constraints of the underlying databases. A.4

Verification Procedure.

Although our dataset is generated through a multi-agent synthesis pipeline, we incorporate a rigorous verification procedure to ensure the quality and reliability of the synthesized documents.

finance

medical

education

Figure 7: Representative examples from the finance, healthcare, and education domains. Each case pairs a multi-table relational schema with a domain-specific synthesized document, where colored links illustrate the grounding between database records and document evidence. First, our framework includes a dedicated Validator Agent that performs automatic verification from two perspectives. At the fine-grained level, it checks whether generated sentences and paragraphs faithfully cover the information contained in the source databases, preventing missing or incomplete evidence. At the database reconstruction level, it evaluates whether the complete database can be accurately recovered from the generated document. Any detected inconsistencies or missing information trigger an iterative repair process to refine the generated documents. Detailed verification and repair procedures are provided in Appendix E.4. Second, we conduct human verification on the generated documents. We recruit seven PhD candidates in computer science, with each reviewer responsible for one specific domain. Each reviewer evaluates the synthesized documents from two aspects: (1) whether the document style and content align with realistic domain-specific documents encountered in practice; and (2) whether the underlying database information can be reliably reconstructed from the generated document. Any cases that fail these criteria, such as unrealistic document patterns, missing database evidence, or inconsistent entity relationships, are filtered out to ensure the quality and reliability of the final dataset. A.5

Dataset Visualization.

Figure 7 presents representative examples from the finance, healthcare, and education domains, each pairing a multi-table relational schema with a realistic domain-specific document. For example, in the finance case, company information is stored in the Companies table, quarterly revenue and net income in Quarterly_Financials, and closing prices in Stock_Prices. The synthesized equity research report integrates these distributed records into statements such as “Acme Corp reported revenue of $125.0 million and net income of $18.2 million,” while linking them through the shared company_id. Recovering the database therefore requires not only extracting individual values, but also resolving entity references, normalizing expressions such as 125,000,000 and “$125.0 million,” and correctly assigning evidence to multiple related tables. Similar examples across the other

Figure 8: Challenging examples of relation extraction that require different inter-table capabilities. domains illustrate the diversity of schema structures, document styles, and cross-table dependencies covered by Doc2DB-Bench.

B

Details of dual-level Taxonomy

Pillar 2 inter-table capabilities are organized into the following four types: Relationship Linking (RL). This capability is required when natural language narratives must be parsed and mapped into structured relational tuples, accurately aligning multiple entities across tables based on cardinality and business rules. We categorize this linking into three subcategories. One-to-Many Allocation. The document provides a compressed statement (e.g., a manager taking over multiple projects), requiring the model to expand it into multiple normalized relationship rows that preserve the underlying 1:N or N:1 cardinality rather than collapsing them. As illustrated in Fig. 8 (a), a statement assigning Alice to Projects X, Y, and Z must be expanded into three separate relationship tuples. Multi-Entity Binding. The document describes intertwined N-ary relations (e.g., distinct suppliers delivering different parts to various projects) in a dense narrative, requiring exact triplet or multientity alignment without mismatching pairs or hallucinating spurious links based on text proximity. Conditional Linkage. Requires evaluating existing table attributes against implicit business rules to automatically establish foreign key relations, even without explicit entity name mentions. Identity Resolution (IDR). This capability is required when entity references within the document are ambiguous, pronominal, or use aliases, necessitating cross-referencing with table attributes or surrounding context to ground the mention to a unique, exact database record. Entity Disambiguation. The document uses an ambiguous alias or name that matches multiple database records, requiring contextual clues and schema attributes to identify the correct entity. As shown in Fig. 8 (b), the alias “Lao Zhang” must be resolved to the employee in the technical department based on the surrounding “technical issue” context.

Coreference Resolution. The target entity is referred to using pronouns, role titles, or implicit aliases scattered across sentences, requiring cross-sentence contextual reasoning to correctly resolve these coreferences to the anchor entity and populate the corresponding relationship fields. Structural Reasoning (SR). This capability is required when relational links cannot be directly extracted from a single text span but must be logically deduced by chaining multiple implicit relations, table structures, or temporal events. Transitive Inference. The document and existing tables provide linked but disjointed relational pairs (e.g., entity A is linked to B, and B is linked to C), requiring the deduction of implicit transitive relations to populate the target schema. Fig. 8 (c) illustrates this case: “Alice leads the Mobile Team” and “the Mobile Team manages Project X” must be composed to recover the implicit relation between Alice and Project X. Global Aggregation. The attributes required to form a complete multi-dimensional relationship are scattered across disjointed text spans, requiring multi-hop reading comprehension to logically chain these spans and assemble all necessary foreign keys into a single valid record. Dynamic Change. The document narrative involves cross-table event timelines, sequential actions, or entity state changes over time (e.g., completing a prerequisite course before a new enrollment), requiring reasoning over these temporal dynamics to determine the final, valid relationship status. Integrity Constraint (IC). This capability is required when the extraction of relationships is governed by negative evidence, exceptions, or boundary rules, demanding that the model suppress naive co-occurrence extraction and strictly adhere to database integrity. Null Relation Extraction. The document provides explicit negative evidence, such as cancelled events or unmet prerequisites, requiring the model to identify an empty relation state (e.g., outputting NULL) and avoid hallucinating links from superficial keyword co-occurrence. Mutual Exclusion. The document specifies broad relationships alongside explicit exceptions or negative constraints, requiring the model to infer relationship boundaries and exclude invalid relational tuples. As illustrated in Fig. 8 (d), although all managers are generally assigned to Project X, the explicit exception “except Bob” requires suppressing the tuple (Bob, Project X).

C

Experiment Setup

C.1

Metric Design Principles: Cell Alignment for Evaluation

Naive sequential matching aligns predicted rows to ground-truth rows by position, which is fragile in two ways. Cross-Tuple Hijacking: a model extracts the correct value but assigns it to the wrong row, penalizing an otherwise valid prediction. Cascading Misalignment: a single missed or inserted row shifts all subsequent alignments, inflating both false-positive and false-negative counts. To address both failure modes, we adopt a Global Maximum Weight Matching strategy over a bipartite graph of ground-truth and predicted rows. Notably, We do not use primary keys as alignment anchors because database-specific identifiers, particularly surrogate keys, are often absent from real-world documents and must be generated during database construction. Different models may therefore assign different identifiers to otherwise equivalent records. Primary-key-based matching would conflate arbitrary identifier generation with extraction correctness. We instead align rows using shared non-key attributes and evaluate key validity separately through database-integrity metrics. ∗ } and 𝑇 = {𝑟 , . . . , 𝑟 } denote the ground-truth and predicted row sets, respecLet 𝑇 ∗ = {𝑟 1∗ , . . . , 𝑟 𝑚 1 𝑛 tively. Matching proceeds in four stages.

Stage 1: Global Scoring Matrix. Let K be the shared column names between 𝑇 ∗ and 𝑇. For each pair (𝑟 𝑖∗ , 𝑟 𝑗 ), we compute a composite similarity score: 𝑠(𝑟 𝑖∗ , 𝑟 𝑗 ) =

) ( 1 ∑ sim 𝑟 𝑖∗ [𝑘], 𝑟 𝑗 [𝑘] , |K | 𝑘∈K

where sim(·, ·) returns 1 for exact numeric equality after normalization and uses normalized Levenshtein similarity for textual values: simtext (𝑎, 𝑏) = 1 −

𝑑Lev (norm(𝑎), norm(𝑏)) ( ), max |norm(𝑎)|, |norm(𝑏)|

where 𝑑Lev denotes the Levenshtein edit distance and norm(·) applies lowercasing, whitespace normalization, and punctuation normalization. Missing or incompatible values receive a score of 0, yielding a global 𝑚 × 𝑛 scoring matrix. Stage 2: Global Sorting. All pairs with 𝑠(𝑟 𝑖∗ , 𝑟 𝑗 ) ≥ 𝜏 (𝜏 = 0.3) are sorted globally in descending order, ensuring high-confidence pairs are resolved first and cannot be preempted by lower-scoring candidates. Stage 3: Double-Locking Assignment. Pairs are assigned greedily from the ranked list. Once (𝑟 𝑖∗ , 𝑟 𝑗 ) is matched, both indices are locked and excluded from further assignment, enforcing strict one-to-one mapping and preventing score inflation via duplicate row generation. Stage 4: Cell-Level Settlement. Within each matched pair, scoring proceeds cell-by-cell as defined in Section 5. Unmatched ground-truth rows contribute all non-empty cells to the missing count; unmatched predicted rows contribute to the hallucination count, ensuring every cell is accounted for exactly once. C.2

Prompts for LLM Score.

LLM-as-a-Judge: Database Extraction Evaluation Prompt You are a database evaluation expert. Please evaluate the quality of database tables extracted from documents. Evaluation Dimensions: • Accuracy: Semantic consistency, exact numeric matching (including decimal places), and string similarity ≥ 90%. • Completeness: Presence of all key information from the standard answer and structural integrity. • Standardization: Compliance with Schema definitions and data type/format consistency. Scoring Constraints: • Numeric values must be exactly equal; string similarity below 90% is a mismatch. • Type mismatches are considered serious errors. • Total score ranges from 0 to 100; a perfect match equals 100 points. Output Format: Provide a comprehensive explanation (≤ 100 words), then output the score as: Evaluation evidence: [explanation] Rating: [[score]]

D

More Doc2DB-Bench Evaluation

D.1

Database Integrity Analysis.

Metric. Beyond attribute accuracy, we evaluate whether models can construct structurally complete and constraint-consistent databases. We measure Entity Coverage and Relation Coverage to assess #Relationship Coverage (RC)

sh pro 4-6 sh ax -4o 5.4 gpt gpt- i2.5-fla ini2.5- -opus- en3-m s-v4-fla d in gem ude qw m e a g cl

1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2

#Reference Integrity (RRIR)

Score

Score

Score

#Entity Coverage (EC) 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2

-4o pt-5.4 -flash .5-pro us-4-6 3-max -flash g i2.5 ini2 -op n v4 in m qwe dsde gem ge clau

gpt

1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2

-4o pt-5.4 -flash .5-pro us-4-6 3-max -flash g i2.5 ini2 -op n v4 in m qwe dsde gem ge clau

gpt

Figure 9: Database-level performance across Entity Coverage, Relation Coverage, and Reference Integrity Rate.

the recovery of entities and inter-entity relationships, respectively. We further report Reference Integrity Rate, which measures the proportion of foreign-key references that correctly point to existing entities, reflecting the validity and consistency of the constructed database. Result. Database-level results show that reconstruction requires both structural coverage and valid cross-table references. Entity Coverage exceeds Relation Coverage by 12.17–24.52 points across all models, highlighting the greater difficulty of recovering inter-table relationships. Coverage also does not guarantee validity: Gemini-2.5-Pro achieves strong coverage but the lowest RRIR, revealing frequent cross-table reference errors. GPT-5.4 leads all three metrics, demonstrating the strongest ability to recover relational structures while preserving database consistency. Finding 4. Database construction requires more than accurate extraction: maintaining relational consistency and valid entity references remains a critical challenge for LLMs. D.2

Full Results of Different Domains

Table 6: Results on DOC2DB-BENCH across various domains. Ver./Size shows model version or size.

Methods

Ver. /Size

Domain Edu.

Overall Acc.

Fin. Med. Uni. Sports Trans. Others Open-source Models

Qwen2.5

14B-Ins 13.25 34.21 37.35 36.44 18.16

47.31

34.82

32.97

LLaMA-3.1

70B-Ins 10.94 13.18 16.29 21.54 27.86

7.30

8.16

14.29

Qwen2.5

72B-Ins 30.41 34.79 40.98 47.73 31.87

38.80

29.98

36.07

Proprietary Models GPT-4o

-

47.89 57.49 54.20 72.32 62.75

68.02

51.11

59.05

GPT-5.4

-

67.67 68.65 66.68 74.78 82.24

85.22

78.49

75.25

Gemini 2.5

Flash

57.63 64.04 41.00 73.13 71.22

78.95

64.57

65.99

Gemini 2.5

Pro

69.81 65.46 76.53 79.75 69.67

74.78

67.94

70.81

Claude Opus

4-6

67.01 65.38 75.51 80.01 78.36

87.83

69.89

73.60

Qwen3

Max

52.06 62.55 56.37 72.15 57.24

70.73

64.10

52.80

DeepSeek-V4

Flash

53.66 59.39 48.23 55.60 57.79

73.56

51.80

57.06

The results reveal a substantial performance gap between open-source and proprietary models. Among open-source models, Qwen2.5-72B achieves the highest overall accuracy of 36.07, but still trails GPT-5.4 by 39.18 points. GPT-5.4 obtains the best overall accuracy of 75.25 and leads in finance, sports, and other domains. Nevertheless, no single model dominates across all domains: Gemini 2.5 Pro performs best in education and medicine, while Claude Opus 4-6 leads in university and transportation, suggesting that different domains place distinct demands on schema interpretation, cross-table reasoning, and domain-specific information normalization. Domain-level performance can be further understood from the normalized dataset statistics. Averaged across all evaluated models, transportation achieves the highest accuracy (63.25), followed by university (61.35), whereas education obtains the lowest accuracy (47.03). Education contains the largest number of rows per database (276.7), together with 8.0 tables and 4.0 relation tables on average, requiring models to recover a relatively large number of tuples while preserving cross-table associations. Finance is also challenging, but primarily because of its relational density rather than its absolute size: each database contains 6.2 tables and 3.9 relation tables on average, with the highest relation-to-entity ratio of 1.70. Medical databases present a different difficulty, having the largest average numbers of columns (66.3) and cells (1,174.0) per database, which increases the burden of broad attribute coverage. In contrast, although transportation documents are the longest on average (53,091 tokens), their databases contain only 3.6 tables and 1.2 relation tables per database, suggest-

ing that document length alone is not the dominant source of difficulty. Overall, the results indicate that performance is more strongly influenced by the interaction of tuple volume, schema width, relation density, and implicit cross-table dependencies than by document or database scale in isolation.

E

Implementation of Doc2DB

During the entire Controllable DB2Doc Reverse-Synthesis process, we design specific prompts and rigorous verification protocols across four key agentic modules to ensure generation fidelity and relational complexity. Firstly, in the capability assignment stage, the Labeling Agent acts as a semantic router. We construct prompts to strategically assign fine-grained intra-table (Pillar 1) and inter-table (Pillar 2) capability labels, which directly control the downstream extraction difficulty. Secondly, during the evidence decomposition stage, the Refiner Agent processes the annotated tables into atomic evidence. To guarantee the mathematical and semantic rigor of this step, we implement a strict checklist-based verification protocol. We design prompts that explicitly instruct the model to verify four critical dimensions: Value Correctness, Label Faithfulness, Schema Linkage, and Null Faithfulness. Thirdly, to bridge the gap between discrete evidence and cohesive natural language, the Writer Agent executes the document generation. We adopt a context-aware generation paradigm where the prompt incorporates a sliding window of historical text and configurable stylistic profiles to emulate realworld document morphological complexity Finally, in order to obtain a high-quality, hallucinationfree corpus, the Validator Agent conducts a rigorous dual-validation pass. We construct prompts to evaluate Coverage Verification and Extraction Consistency. Documents that fail these checks trigger an iterative reject-and-rewrite mechanism. E.1

The prompt for Labeling Agent

Pillar 1 Assignment Prompt

Pillar 2 Assignment Prompt

Goal: Assign intra-table (Pillar 1) capability labels to target cells in the entity table. Inputs: • Input Table: {markdown_table} • Non-target Key Cols: {non_target_key_columns} • Attribute Schema: {attribute_descriptions} • Capability Definitions: {capability_definitions} • Target Label Ratio: {label_ratio} Instructions: 1. Assign labels that increase single-table extraction difficulty while preserving correctness. 2. Type-Strategy Compatibility (CRITICAL): Independently analyze the semantic nature and data type of each attribute. Only assign strategies logically/mathematically suitable for the cell’s value. 3. A cell may have 0, 1, or multiple strategies (default is none). 4. Assign creatively but ensure text remains fluent. Avoid over-complicating simple facts. 5. DO NOT assign labels to Non-target Key Columns. 6. For composite primary keys, use ", " as separator. 7. Label coverage: at least the target ratio of assignable cells must have non-empty labels. 8. Return JSON only. Output JSON Schema: {"assignments": {"<pk_1>": {"<attr_1>": []} } }

Goal: Assign inter-table (Pillar 2) capability labels to target cells in the relation table. Inputs: • Input Table: {markdown_table} • Table Schema Context: {relation_context} • Attribute Schema: {attribute_descriptions} • Capability Definitions: {capability_definitions} • Target Label Ratio: {label_ratio} Label Assignment Constraints: • IDR_ED, RL_O2M, RL_MB, and RL_MI must each appear alone. Do NOT combine them. • IDR_ED, RL_O2M requires that the same foreign key entity appears in ≥ 2 rows. Do NOT assign if unique. Instructions: 1. Assign labels that increase inter-table linking, topology, and structural extraction difficulty. 2. Topology-Strategy Compatibility (CRITICAL): Analyze structural distribution. Only assign relational strategies logically suitable for the topology. 3. Output ONLY row-level labels per relation row_key. Do NOT output per-attribute labels. 4. A row may have 0, 1, or multiple strategies (default is none). 5. Top-level key MUST use the relation table PK string. 6. Composite keys format: <pk1>, <pk2> (e.g., 12, 10). 7. Label coverage: at least the target ratio of rows must contain non-empty labels. 8. Return JSON only. Output JSON Schema: {"assignments": {"<pk_1>": [], "<pk_2>": []} }

The prompt above illustrates how the Labeling Agent acts as a semantic router to assign fine-grained intra-table (Pillar 1) and inter-table (Pillar 2) capability labels. It explicitly instructs the model to analyze data types and topological contexts, ensuring that specific reasoning challenges are injected logically to control downstream extraction difficulty while adhering to target label ratios. E.2

Refiner: Principle of Checklist

Given the annotated targets, the Refiner decomposes each target into atomic evidence that realizes its assigned capability labels, forming a global evidence pool for subsequent document generation. When multiple labels are assigned, the evidence must jointly satisfy the corresponding intra-table and inter-table reasoning requirements. To ensure the rigor of the synthesized evidence, we deploy an LLM-as-a-verifier [50] using a strict checklist-based protocol. The verification assesses the generated evidence guidance across four critical dimensions: • Value and Relation Correctness: Verifies that the guidance accurately reflects target cell values and inter-record relations, ensuring numerical correctness for calculated fields and allowing complete relational tuples to be unambiguously recovered. • Label Faithfulness: Verifies that the guidance faithfully instantiates the assigned capability labels and rejects cases that omit or misrepresent the intended reasoning requirements. • Schema Linkage: Ensures that the guidance is grounded in valid schema elements and preserves the intended structural dependencies among tables. For relational tables, primary- and foreign-key references must be correctly identified and linked, so that participating entities can be unambiguously connected and the resulting evidence remains consistent with the target database schema. • Data Integrity & Null Faithfulness: Ensures that the guidance contains only information supported by the target database and does not introduce values from unrelated cells or records. It also verifies that missing, unavailable, or non-applicable attributes remain uninstantiated, preventing the generation of unsupported values or spurious relational links. On failure, the verifier acts as a critic, returning a specific rationale and actionable revision suggestions (e.g., “Instruct the writer to explicitly mention both the supplier and project names”). This process iteratively refines the text until it passes all checks or reaches a retry limit. E.3

The prompt for Writer Agent

Writer Agent Prompt [System Role & Objective] You are an expert Writer Agent operating within a parameterized document synthesis pipeline. Your goal is to write ONE coherent, polished, and natural-sounding section of a larger document while perfectly preserving structured ground-truth facts. [Generation Context] • Previous Block Context (Tail Anchor): {previous_context} • Current Block ID: {block_id} (Index: {block_index} of {total_blocks}) [Base Parameters] • Global Target Document Length: {document_length_tokens} tokens • Suggested Token Budget for THIS Block: {target_block_tokens} tokens • Document Style & Tone: {document_style} • Section Template Hint: {section_template} [Injection Protocols] • Hard Cases Protocol: {hard_cases_protocol} • Controlled Noise Protocol: {noise_protocol} • Representation Complexity Protocol: {complexity_protocol} [Mandatory Ground Truth (Evidence)] You MUST explicitly embed the following structured facts. Do NOT drop any fact: {facts_with_tags} [Execution Instructions]

1. FACTUAL FIDELITY: Embed all mandatory facts without altering their meaning. 2. TAG PRESERVATION: Keep facts wrapped in their original tags (e.g., <rX_cY>). 3. STYLISTIC ADAPTATION: Match the requested style, tone, and template. 4. PROTOCOL COMPLIANCE: Strictly follow any active injection protocols; ignore empty ones. 5. LENGTH & PACING: Stay close to the {target_block_tokens} budget. 6. SEAMLESS BRIDGING: Continue logically and smoothly from the previous context. 7. FORMATTING: Use natural prose paragraphs. No bullets, markdown, or JSON. 8. OUTPUT STRUCTURE: Start exactly with the Section Title, followed by the body. 9. FINAL OUTPUT: Return ONLY the synthesized text, without conversational filler.

The prompt above demonstrates how the Writer Agent transforms serialized atomic evidence into cohesive, long-form natural language documents. It guides the model to employ a context-aware generation paradigm, seamlessly weaving discrete facts into fluent narratives while conforming to designated stylistic profiles and mimicking the morphological complexity of real-world corpora. E.4

The prompt for Validator Agent

Verification Agent Prompt

Repair Agent Prompt

Your Goal: Verify whether the generated text is faithful to the table content without fabrication.

Input:

Table Content: {markdown_table} Generated Text: {generated_text} Verification Checks: To ensure Cell Extraction Consistency, Key values stated in generated_text must be extractable or inferrable from Table Content. Output Format: Respond with ONLY a single, valid JSON object: { "ok": true/false, "errors": [ { "description": "Error description", "suggestion": "How to fix" } ] }

Goal: Repair a section of text that failed verification. • Original Text: {content} • Errors: {errors} • Required Facts: {facts} Instructions: 1. Repair the section to fix the reported errors. 2. Ensure all required facts remain correct. 3. Keep the narrative flow natural. 4. Preserve all provenance-style tags already present in the original text. Output Format: Return ONLY the repaired text content. No titles, no metadata, no markdown code blocks.

FINAL INSTRUCTION: Output ONLY the valid JSON object. If any check fails, set ok to false and include all issues in errors.

The prompt above details the rigorous dual-validation protocol executed by the Validator Agent to guarantee the fidelity and relational consistency of the synthesized documents. It tasks the model with strictly evaluating coverage verification and extraction consistency, ensuring no evidence is omitted or hallucinated, which inherently serves as the trigger for our iterative reject-and-rewrite mechanism.

F

Authenticity Verification

F.1 LLM & Human Score The synthesized documents in DOC2DB-BENCH should be fluent, logically coherent, and faithful to the conventions of real-world domain documents rather than appearing as artificial or templatebased constructions. To evaluate document quality and authenticity, we adopt a four-dimensional framework combining LLM-as-a-judge with human evaluation. As shown in Table 6, the framework assesses lexical richness, logical consistency, textual coherence, and genre fidelity, each assessed using a five-point rubric, following a similar strategy of previous work (e.g., LLMs4Synthesis [3] and

Table 7: Evaluation rubrics for generated documents (1=worst, 5=best). Score Lexical Richness

Logical Consistency

Textual Coherence

Genre Fidelity

1

Repetitive: Minimal variety, robotic repetition

Fragmented: No connections, random claims

Incoherent: Difficult to follow, random jumps

Mismatch: Incorrect genre conventions

2

Limited: Simple vocabulary, narrow range

Weak: Loose transitions, vague reasoning

Poor flow: Jarring transitions, disconnected

Weak: Limited and inconsistent cues

3

Acceptable: Adequate variety, standard usage

Acceptable: Clear order, basic signaling

Acceptable: Some awkward transitions

Plausible: Recognizable but generic style

4

Versatile: Natural synonyms, precise terminology

Compelling: Strong arguments, coherent progression

Smooth: Clear progression, minor issues

Authentic: Consistent genre-specific conventions

5

Sophisticated: Rich nuances, professional mastery

Rigorous: Flawless chain, seamless attribution

Seamless: Natural flow, effortless transitions

Professional: Realistic and polished genre presentation

Figure 10: menu case DTBench [51]). We use DeepSeek-V3.2 as the automatic evaluator. In addition, five graduate students with research experience in document intelligence independently assess 50 randomly sampled documents using the same rubrics. The LLM and human evaluation results in Table 5 consistently demonstrate the high linguistic quality and domain authenticity of the synthesized documents. F.2 Real Case Comparison Fig. 10 – 13 provide qualitative comparisons between synthesized and real-world documents across diverse domains, including finance, aviation, media, and restaurant menus, further demonstrating the quality and realism of our synthesis pipeline.

Figure 11: finance case

Figure 12: airline case

Figure 13: twitter case

Related documents

• A MODEL FOR HISTORICAL DATABASES#941609
• LitCurate: A Configuration-Driven AI-Assisted Framework for Scientific Database Construction with an Application to Lower-Mantle Equation-of-State Data#780931
• A Fair Benchmarking of Deep Relational Database Learning Models#343580
• STUDIUM.HTR.NER.NEL: A Pipeline for Handwritten Text Recognition, Named Entity Recognition, and Named Entity Linking#670792
• Strategies for Introducing Databasing into Science.#333322