arXiv:2606.30371v1 [cs.DB] 29 Jun 2026
MaDI-Bench: An End-to-End Data Integration Benchmark Aaron Steiner*
Ralph Peeters*
Christian Bizer
Data and Web Science Group University of Mannheim Mannheim, Germany [email protected]
Data and Web Science Group University of Mannheim Mannheim, Germany [email protected]
Data and Web Science Group University of Mannheim Mannheim, Germany [email protected]
Abstract—Data integration combines heterogeneous data sets into a single, coherent representation. Data integration involves a sequence of interdependent tasks including schema matching, value normalization, entity blocking, entity matching, and data fusion. Existing benchmarks either evaluate these steps in isolation or cover only incomplete versions of the data integration pipeline, omitting specific steps. The lack of public end-toend data integration benchmarks hinders research on data integration methods that address the integration process as a whole. This paper fills this gap by introducing the Mannheim Data Integration Benchmark (MaDI-Bench), the first benchmark for the end-to-end integration of relational tables covering all steps of the integration process. MaDI-Bench contributes (i) a set of base end-to-end data integration tasks spanning several application domains, each requiring the full schema matching, value normalization, entity matching, and conflict resolution pipeline; and (ii) a generic method for deriving task variants that mitigates rapid benchmark saturation as data integration systems advance. We validate the benchmark using human-engineered pipelines, a best-of-breed pipeline, and an LLM-based pipeline. The validation demonstrates the utility of the benchmark for measuring the step-wise as well as the end-to-end performance of data integration pipelines. All benchmark artifacts are available for public download. Index Terms—data integration, data lakes, schema matching, entity matching, data fusion, end-to-end evaluation, large language models
I. I NTRODUCTION Data integration [1], [2] combines data from multiple heterogeneous sources into a single, coherent representation. Data integration involves several interdependent steps: matching the schemata [3] of the input datasets, normalizing values to a common format, identifying records that refer to the same real-world entity (entity matching [4]), and resolving conflicts between the values of matching records into a single fused value per attribute (data fusion [5]). Each step has been studied in separation for decades [1], [2], but in practice, these steps are tightly coupled: the output of one step constrains what the next step can achieve, and errors propagate through the pipeline. For example, successful value normalization makes the following entity matching task easier, unsuccessful entity matching results in multiple record clusters per real-world entity or overly large clusters describing multiple real-world * These authors contributed equally to this work.
entities. Both types of errors impact data fusion, as multiple clusters might end up as duplicate records in the output and as over-sized clusters covering different entities make conflict resolution in data fusion difficult. Therefore, evaluating the steps of the data integration process in isolation gives an incomplete picture of how a data integration system performs on the end-to-end data integration task that practitioners actually face. Recent progress in large language models (LLMs) and agentic systems is changing what is feasible: LLM-based agents can read heterogeneous inputs, propose schema mappings, normalize values, and develop conflict resolution rules without human involvement [6]. Agents can also collaborate with human data engineers using human-in the loop approaches [7]– [9]. This makes automated and human-AI end-to-end data integration a realistic target. To drive and measure progress towards this goal, the community needs benchmarks that evaluate the whole integration pipeline, not just one step at a time. Existing benchmarks do not provide for this need: Many benchmarks target single data integration steps, e.g. schema matching [3], [10], entity matching [4], [11], and fusion [5], [12] are each evaluated on their own datasets with step-specific metrics. ETL and ELT benchmarks such as TPC-DI [13] and ELT-Bench [14] cover end-to-end transformation and loading workloads, but not end-to-end data integration tasks involving entity matching and conflict resolution. What is missing is a benchmark whose tasks span the full relational data integration pipeline (schema matching, value normalization, entity matching, and data fusion), that provides tasks at controlled difficulty levels across several domains, and that supports the measurement of reference-free structural end-to-end metrics as well as ground-truth-based end-to-end metrics. This paper presents the Mannheim Data Integration Benchmark (MaDI-Bench), an end-to-end benchmark for evaluating data integration systems. MaDI-Bench provides integration tasks that take several heterogeneous source tables as input and require their integration into a single, consistent target table as output, exercising schema matching, value normalization, entity matching, and data fusion together. The MaDI tasks cover five application domains: Games, companies, music, products, and scientific papers. Each task includes a fusion
validation set and a fusion test set containing together 11,000 human-validated ground truth values. We also introduce a variant-generation method to provide additional easy, medium, and hard variants for each baseline task, so that integration methods can be tested for effectiveness and efficiency on tasks mirroring the different levels of difficulty that can appear in real-world settings. Overall, MaDI-Bench offers 20 unique integration tasks across the five domains. To assess pipeline output, we organize metrics along three quality dimensions (coverage, consistency, and correctness) and three evaluation setups (reference-free structural, silver-standard, and groundtruth), including structural metrics such as entity gain, density gain, output density, and a fusion ratio. We validate MaDIBench using human-engineered PyDI1 pipelines, an LLMbased data integration workflow [6], and a best-of-breed pipeline that tests multiple approaches from the literature per pipeline step with the final pipeline consisting of the chained composition of the per-stage winners. This paper makes the following contributions: 1) A set of base end-to-end data integration tasks across five application domains (games, companies, music, products, and scientific papers), each spanning the subtasks schema matching, value normalization, entity matching, and data fusion. We provide ground truth in the form of validation and test sets for each subtask as well as for the end-to-end workflow. 2) A variant-generation method for deriving end-to-end data integration tasks at different levels of difficulty from the five base tasks. The variants of increased difficulty aim at making the benchmark more future-proof as data integration systems improve, while the easy variant supports the evaluation of simple but computationally efficient methods. 3) We validate MaDI-Bench using different end-to-end pipelines: human-engineered pipelines, a LLM-based workflow, and a best-of-breed pipeline which chooses between alternative methods for each step of the integration process. Resource Availability: All artifacts together with the code for replicating the experiments are available in the MaDIBench repository2 . Additional details and examples can be found on the benchmark’s website3 . II. T HE BASE TASKS MaDI-Bench consists of five end-to-end data integration tasks: Games, Companies, Music, Products, and Scientific Papers. Each of these base tasks requires an integration system to solve the full pipeline, from schema matching and value normalization through blocking and entity matching, to data fusion. As input, a system receives a set of source tables, metadata describing the tables, the target schema with its constraints and taxonomies, and optionally labeled training and 1 https://github.com/wbsg-uni-mannheim/PyDI/tree/main 2 https://github.com/wbsg-uni-mannheim/MaDI-Bench 3 https://wbsg-uni-mannheim.github.io/MaDI-Bench/
validation sets. It must return a single fused table that conforms to the target schema and contains one record per real-world entity. For the step-wise evaluation, a system additionally has to report two intermediate results: the proposed mapping from source attributes to the target schema and the matched record pairs. These are evaluated using the gold schema mapping and the holdout test sets. Figure 1 gives an overview over the inputs, the data integration tasks, and the benchmark artefacts. The artefacts and benchmark tasks are described in the following sections. Table I provides key statistics and an overview of the schemata of the datasets of the base tasks.
Fig. 1. End-to-end data integration workflow and benchmark artefacts.
A. Benchmark Artefacts Each benchmark task provides the source data, descriptions of that data, and the resources to evaluate every step of the pipeline. Across the five base tasks, these artefacts amount to more than 93,000 labeled record pairs for entity matching, 1,000 human-verified fusion records carrying close to 10,000 verified attribute values, and a gold schema mapping per task. Source tables and metadata: Each task provides one table per source. Every table is accompanied by a metadata file that records provenance information of the data, the columns it contains, and the date it was published. The source publication dates span more than a decade, from the 2014 Forbes Global
TABLE I S TATISTICS AND ATTRIBUTES OF THE M A DI BASE TASK DATASETS . I DENTIFIER COLUMNS ARE EXCLUDED FROM ATTRIBUTE AND DENSITY COUNTS . OVERLY LONG ATTRIBUTE LISTS ARE TRUNCATED WITH [...]. Task
Games
Source
Rows Attrs. Str Num Date Density Attributes
DBpedia Metacritic Sales
46,580 20,494 7,877
6 8 10
5 5 6
0 2 3
1 90.9% title, launch yr, studio, system, genre, franchise 1 97.7% game title, year published, made by, console, genres, press rating, player rating, age rating 1 98.7% prod title, launch dt, studio, dist, hw, genre, press score, comm rating, age classification, units sold mm
–
10
7
2
1
DBpedia
10,085
8
5
2
Forbes FullContact
2,000 1,931
6 5
4 4
2 0
1 62.2% org name, established, nation, headquarters, sector, keypeople name, total assets val, annual income 0 99.1% company, url, region, business segment, asset value, sales figure 1 61.9% Attribute 2, Attribute 3, Attribute 4, Attribute 5, Attribute 6
Target schema
Companies
Music
Target schema
–
8
5
2
1
Discogs Last.fm MusicBrainz
22,627 9,865 4,763
8 4 6
6 3 4
1 1 1
1 98.4% title str, performer, pub dt, origin loc, duration, imprint, category, tracks track-name 0 86.7% album title, band, album length, tracks track-name 1 96.3% Attribute 2, Attribute 3, Attribute 4, Attribute 5, Attribute 6, Attribute 9
Target schema
–
8
6
1
1
Dataset 1
812
25 16
9
Dataset 2
812
25 16
9
Dataset 3
762
25 16
9
Dataset 4
626
25 16
9
0 55.0% manufacturer, product name, product description, list price, currency code, product url, name and description, model name, manufacturer part number, category, gpu chipset, video memory gb, [...] 0 54.3% brandName, name, descriptionText, priceAmount, currency, productUrl, titleAndDescription, modelName, mpn, productCategory, chipset, vramGb, capacityGb, readSpeedMbps, writeSpeedMbps, busType, interfaceType, [...] 0 53.7% Brand, ProductTitle, Details, Price, Currency, Link, TitleDetails, Model, PartNo, Type, Chipset, MemorySizeGB, [...] 0 54.2% mfr, name, desc, amt, cur, link, name desc, mdl, pn, cat, chip, vram, cap gb, rd mbs, wr mbs, bus, iface, [...]
–
25 16
9
0
Crossref
60,749
13 10
2
DBLP
60,591
9
8
0
OpenAlex
60,719
12
9
2
1 85.9% work type, title text, contributor names, issued year, container title, publisher name, abstract text, volume id, issue id, page first, page last, reference total, cited total 1 68.1% entry type, publication title, author list, pub year, venue name, volume no, issue no, page start, page finish 1 88.6% work kind, display title, authors list, year published, source name, topic terms, volume tag, issue tag, start page, end page, refs count, citations count
–
11
8
2
Products
Target schema
Papers
– name, releaseYear, developer, genres, publisher, platform, criticScore, userScore, ESRB, series
Target schema
1
– name, founded, country, city, industry, assets, revenue, keypeople
– name, artist, release-date, release-country, label, genre, tracks, duration
– brand, title, description, price, priceCurrency, url, title description, model, model number, product type, chipset name, vram gb, storage gb, read speed mb s, write speed mb s, bus type, interface type, [...]
– type, title, authors, publication year, journal, volume, issue, first page, last page, referenced works count, cited by count
2000 list to 2026 snapshots of DBLP, Crossref, and OpenAlex. We deliberately include dated sources, since outdated values are a recurring challenge in real integration settings. An advanced system can use the metadata as part of the integration, for example, to weigh a source’s trustworthiness during fusion from its provenance and publication date. Target schema: Defines the columns that the output table (Table I) should contain. In addition to the column names, each attribute is given a value constraint that bounds its values, for example a company founding year between 1800 and 2026. Each categorical attribute is tied to an established taxonomy that ships with the benchmark, such as the Global Industry Classification Standard4 for industries, and the schema fixes the taxonomy level at which values should be fused. Gold schema mapping: Each task includes a gold mapping from the source attributes to the target schema, against which the correspondences proposed by a system are scored during the evaluation of the schema matching step. Entity matching sets: Each task provides labeled training, validation, and test sets for entity matching. Table II reports 4 https://www.msci.com/indexes/index-resources/gics
the number of correspondences (record pairs) between sources (e.g., MusicBrainz-Discogs and MusicBrainz-Last.fm for the Music task) and the share of positive pairs (matches) in the test set, which is between 25% and 33% across tasks. The sets concentrate on difficult to match corner cases near the decision boundary between matches and non-matches: true matches with dissimilar surface forms, and non-matches with high surface similarity [15]. To reduce selection bias, candidate pairs are drawn from several blocking strategies per task, including Jaccard-similarity and key-based blocking. The sets are disjoint at the pair level: no labeled pair appears in more than one set. We verified each test set by hand, except for Papers, whose matches are automatically derived from available DOI identifiers, and Products, which reuses the mapping of the WDC Products [15] benchmark from which the datasets were derived. Entity matching sets are also used for evaluating blocking in MaDI-Bench: training and validation sets can be used to tune blockers for pair completeness and reduction ratio, whereas the test set provides the final evaluation.
TABLE II L ABELED RESOURCES FOR ENTITY MATCHING AND FUSION EVALUATION . Entity matching Task Games Companies Music Products Papers
Src. pairs Train
Val.
Fusion validation
Test Test +% Records
2 934 234 739 2 1,971 948 599 2 36,658 17,010 2,000 3 4,488 600 600 2 10,666 2,666 13,332
30.0% 29.4% 33.3% 25.0% 25.0%
100 100 100 100 100
Fusion test
Values Records Values 899 682 799 1,466 1,033
100 899 100 691 100 770 100 1,479 100 1,041
Fusion validation and test sets: The fusion evaluation sets are created by selecting entities whose records disagree on the values of the entity, ensuring that conflict resolution has to take place. Annotators determined the correct value of each attribute through web research across trustworthy sources that are not part of the task sources. Every gold value is grounded in the actual real value for that entity instead of chosen from the values in the data sources. Disagreements between annotators were resolved by gathering further evidence through a second round of web research. Each task in MADI-Bench provides 100 validation and 100 test records for data fusion. Table II reports the number of target values they carry, excluding identifiers. The validation set allows for integration systems to tune their conflict-resolution strategy, while the test set is used to calculate the final fusion accuracy. The attribute-specific comparison functions that score fusion accuracy, for example, tolerance-based numeric comparison, are a part of the PyDI library. Other implementations can be used, provided they follow the metric definitions in the PyDI documentation5 . The following section introduces each task and its specific challenges. B. Benchmark Tasks Table I summarizes the source-level characteristics of the five base tasks. For each source, we report the number of rows, the number of non-identifier attributes, and the density, measured as the fraction of non-empty cells over these attributes. The target schema row reports the number of non-identifier attributes in the integrated output schema. The attributes column lists the corresponding non-identifier attributes. Product schemas are abbreviated with [...] because they contain a large number of technical attributes. Games: The Games task requires the integration of datasets from three video-game sources: Metacritic, a dataset about game sales, and games data from DBpedia. Metacritic focuses on review scores and ESRB ratings, the sales dataset contains commercial performance information, and DBpedia contributes encyclopedic attributes such as release dates, developers, platforms, genres, and series information. The target schema contains ten non-identifier attributes. The main challenge is that the same game on a different platform is a separate entity, so a matcher must not over-rely on title: identical titles recur across platforms, and sequels, special editions, and downloadable content differ only slightly in name. The 5 https://github.com/wbsg-uni-mannheim/PyDI/tree/main/docs
task also requires the normalization of attribute values into platform and genre taxonomies, as well as the normalization of dates and rating values using different vocabularies. Companies: The Companies task requires the integration of records from the Forbes Global 2000 list, a DBpedia company extract, and a FullContact company-profile sample. Forbes provides financial and ranking-oriented information, DBpedia contributes founding dates, headquarters, industries, and key people, and FullContact adds contact and personnel information. The target schema contains nine non-identifier attributes. The included companies span the globe, making normalization of legal suffixes and resolving their locations non-trivial. The difficulties of the task further include company-name variants, spelling differences, corporate hierarchies, and the normalization of financial figures, countries, and terms for different industry categorization taxonomies. Music: The Music task integrates release-level records from Discogs, Last.fm, and MusicBrainz. Discogs provides release metadata with music labels, genres, countries, and track lists. Last.fm contributes album metadata, and MusicBrainz provides release, artist, and label information. The 37,255 input records describe albums, EPs, and singles, with partial overlap across sources. The target schema contains eight non-identifier attributes. The main challenge lies in heterogeneous value formats, for example, album durations are recorded in different formats across sources. Further difficulties involve title and artist variants, sparse source records, and normalization of dates, countries, and track lists. Products: The Products task is based on a data sample of the entity matching benchmark WDC Products [15]. The product attribute values were extracted from free-text product offers using information extraction techniques. Product datasets cover GPUs, SSDs, HDDs, and USB sticks. The sources describe product offers using different source-schema variants. The target schema contains 25 non-identifier attributes, making it the largest schema in the benchmark. The large schema makes matching delicate: a small difference in a single technical attribute can differentiate a match from a non-match, and the same attribute appears under different names using different value formats across sources. The task also includes normalization taxonomies for units, capacities, dimensions, weights, and speeds. Scientific Papers: The Scientific Papers task integrates a selection of computer-science paper records from DBLP, Crossref, and OpenAlex. Each source annotates papers with bibliographic attributes such as title, authors, publication year, venue, page range, and citation-related counts. The target schema contains 11 non-identifier attributes. As the largest task by row count, Scientific Papers makes blocking efficiency a challenge: a blocker with a poor reduction ratio leads to hundreds of thousands of record pair comparisons. As a result, this task is especially suited for evaluating the efficiency of integration systems in addition to their effectiveness. The challenges of the task involve title and author-list variants, incomplete identifiers, publication-type and venue normalization, and sparse metadata for volume, issue, pages, keywords,
and citation counts. III. E XPERIMENTAL VALIDATION This section validates the MaDI-Bench benchmark using three end-to-end data integration pipelines. Section III-A describes the different pipelines, Section III-B introduces the metrics that we use for the evaluation, and Section III-C reports per-step as well as end-to-end results. A. End-to-End Integration Pipelines We validate the benchmark using three end-to-end data integration pipelines. The pipelines span the design space from completely hand-engineered to fully LLM-driven. Two of the pipelines build on the PyDI data integration framework6 : the human-designed pipeline (P1) and the LLM-based pipeline (P3). Both have access to the same integration functions and differ only in who configures them, a human or an LLM. The best-of-breed pipeline (P2) chooses between different specialized methods from the literature for each task step. (P1) Human-designed PyDI pipeline: P1 consists of specific workflows for each of the five base tasks. The workflows all implemented using the PyDI data integration framework and were manually engineered by master and PhD students of the University of Mannheim. The workflows were reviewed and improved in several iterations. The workflows that achieved the best results are provided in the benchmark’s Github repository. Each workflow follows the same order: The data engineers run the PyDI LLM schema matcher, inspect the generated correspondences, and hand-correct missed or misplaced attributes before translating the sources into the target schema. Afterwards, a normalization step aligns value formats, including date and number parsing, unit conversion, and selected taxonomy mappings. In Games, for example, this includes mapping platform variants to the target taxonomy, y, which has a strong impact on matching and fusion outcomes. Blocking uses either embedding-based blockers or task-specific blocking keys. The validation target for blocking is at least 97% pair completeness, meaning that at least 97% of labeled matches remain in the candidate set. Among blockers that reach this floor, engineers prefer the one with the highest reduction ratio. For entity matching, engineers compare rule-based matchers and train tree-based matchers using similarity features. They further inspect validation errors, and apply global one-to-one matching algorithms to the results if deemed necessary. For fusion, they choose PyDI conflict-resolution heuristics per attribute based on data type, domain knowledge, and validationset behavior. The employed heuristics include voting, source trust, longest or shortest string, numeric aggregation, and set union for list-valued attributes. P1 is a practical human reference point which does not claim optimality. Its fused output also serves as the silver reference for the referencebased metrics in Section III-B. (P2) Best-of-breed pipeline: The best-of-breed pipeline chains strong specialized components per integration step 6 https://github.com/wbsg-uni-mannheim/PyDI
to achieve a good end-to-end integration result. It executes the four integration steps (schema matching, blocking, entity matching, and fusion) without backwards iteration. At each step, a committee of competing methods is executed, the result of every method is scored using the validation set, and the winner’s output is passed to the next step. The final pipeline is the chained composition of the per-step winners and emits a single output table. The schema matching committee consists of COMA [16], which aggregates several label- and instance-based submatchers into a combined similarity score, the Magneto [17] matcher that pairs a small retrieval model with an LLM reranker, a Sentence-BERT embedding matcher [18] based on nearest-neighbor retrieval of column embeddings, and PyDI’s label-, instance-, duplicate-based and LLM schema matchers. The blocking committee consists of standard blocking, token blocking and sorted-neighborhood blocking [4], BM25 blocking [19], Sentence-BERT [18] blocking, and the supervised contrastive blocker SC-Block [20]. The committee keeps, for each pair of source datasets, the blocker that clears a pair completeness floor of 97% at the highest reduction ratio on the entity matching validation set. If that floor is not reached by any committee member, the blocker with the next highest pair completeness is selected. The entity matching committee consists of the PLM-based Ditto [21] and Magellan [22], a matcher that automatically creates and selects similarity-based features for a random forest matcher. The fusion committee consists of PyDI conflict-resolution heuristics and learned truth-discovery models. Each heuristic resolves a conflict by a fixed rule such as majority voting, source trust, longest string, median, or set union. A perattribute selector chooses for each attribute the heuristic that scores best on the fusion validation set [5]. Majority voting and source trust are additional uniform baselines that apply a single rule to every attribute. The remaining members are truth-discovery models that estimate source trustworthiness: TruthFinder [23], LTM [24] AccuSim [25], CASEFusion [26], and FusionQuery [27]. (P3) LLM-based pipeline: P3 is the LLM-based end-toend DI pipeline described in [6]. It uses the same methods from the PyDI framework as the human pipeline, but the LLM configures the pipeline. The LLM performs schema matching by calling the LLM schema matcher. For the other steps it configures non-LLM methods instead of integrating the records itself, allowing the pipeline to scale. The pipeline receives the source tables, a target schema, and the benchmark taxonomies. It matches the schemas using GPT-5.5 and normalizes the values by mapping them to the predefined taxonomies. Non-categorical attributes are normalized by a column profiler that selects a semantically correct normalization function. An embedding-based blocker then reduces the sources to candidate pairs. From a sample of these pairs, the pipeline builds its own machine-labeled entity matching training and validation sets [6], using GPT5.2 as the labeler. These machine-labeled sets are independent of the sets that are provided by this benchmark. Using these
artefacts, the LLM trains an entity matcher, choosing among rule-based matchers and feature-based models such as random forest and XGBoost. For data fusion, the model determines a configuration which assigns each attribute a conflict-resolution heuristic from the set of heuristics implemented in PyDI, for example longest string, voting, or source-based trust.
TABLE III E ND - TO - END METRICS BY QUALITY DIMENSION . R EF. LISTS THE REFERENCE LEVELS AT WHICH A METRIC APPLIES : REFERENCE - FREE (RF), GROUND TRUTH (GT), AND SILVER (S). Metric
Ref.
Coverage Entity Gain
RF
Density Gain
B. Evaluation Metrics The MaDI-Bench test driver calculates both step-level and end-to-end evaluation metrics. For the individual pipeline steps, we use the established metrics: Schema matching is evaluated as the F1 score over predicted attribute correspondences. Blocking is evaluated with pair completeness and reduction ratio. For entity matching we report F1 on the test set. Data fusion is evaluated using accuracy, the share of the fusion values that match the ground truth value given the benchmark’s attribute-specific tolerance ranges. Beyond the step-wise metrics, we calculate end-to-end metrics across the full data integration process, organized along two orthogonal axes: the quality dimension and the reference level. The quality dimension capture three notions of output quality: Coverage measures whether the fused output contains the relevant amounts of entities and values. It combines four reference-free signals computed on the fused output alone with three metrics that compare the output’s entities and values against a reference table. Consistency measures whether the output conforms to the target schema and the expected value formats. Both consistency metrics build on a per-cell schema validity check, reported once on the output alone and once as a difference against a reference. Correctness measures whether the matched records and fused values are actually correct and is therefore defined only against a reference. Its metrics separate cluster-level from value-level agreement: output and reference clusters are first aligned by best record overlap, after which the value metrics are computed on the aligned pairs. The second axis reflects the reliance on reference data for the evaluation. We distinguish three levels of reference: Reference-free metrics characterize the output of a pipeline on its own. They require no labels and are always available, but can only describe the structure of the output, not its correctness. Ground-truth-reference metrics compare against the test sets provided together with the benchmark, which provide the highest level of correctness but are calculated only on the small slice of the output that is covered by the test sets. Silver-reference metrics compare the fused output against the output values of a reference pipeline. In this paper, we use the output of the human-engineered PyDI pipeline (P1, Section III) as silver standard. Using the output of a reference pipeline as silver standard results in broader evaluation coverage than the ground-truth reference, but offers weaker correctness guarantees. Table III defines the individual metrics and the reference level(s) at which they apply.
Output Density Fusion Ratio
Entity recovery Value drift
Value-density ∆ Consistency Schema validity
Definition
Relative change in the number of output records compared to the largest input source. RF Non-null cell density of the fused table compared to the densest input source. RF Fraction of non-null cells in the output table. RF Share of output records assembled from two or more source records, i.e., how much cross-source fusion occurred. GT, S Fraction of reference entities recovered as a single output cluster with exact overlap (Jaccard = 1). GT, S Mean per-attribute distance between output and reference value distributions: Jensen-Shannon [28] for categorical and text, normalized Wasserstein [29] for numeric and date attributes. GT, S Mean per-attribute difference between the output’s and the reference’s number of filled values.
Fraction of filled cells satisfying the target schema’s constraints on their attribute: data type, numeric range, enumeration, string pattern or format, and taxonomy membership. Schema validity ∆ GT, S Output’s schema-validity rate minus the reference’s. Positive values mean the output conforms more closely than the reference. Correctness BCubed precision
BCubed recall
BCubed F1 Fusion accuracy
Fully-correct rate
RF
GT, S Cluster purity: per-record fraction of an output cluster’s members that share the record’s reference cluster, averaged over records. Low values show over-merging. GT, S Cluster completeness: per-record fraction of a reference cluster’s members that land in the record’s output cluster, averaged over records. Low values show over-splitting. GT, S Harmonic mean of BCubed precision and recall, the main measure of clustering agreement. GT, S Unweighted mean of per-attribute value accuracy. A fused cell is correct when it matches the aligned reference value under the attribute’s evaluation function. GT, S Fraction of aligned output clusters whose fused record matches the reference on every evaluated attribute simultaneously, the strictest measure.
C. Validation Results for the Base Tasks We run all pipelines on the five base tasks of MaDI-Bench (Section II). Each pipeline is configured on the provided validation sets. P3 instead uses its own self-generated validation sets. When a component needs training data, P1 and P2 use the benchmark-provided training sets. P3 generates its own training data, following [6]. Each pipeline is run once per task. P1 runs the PyDI LLM schema matcher with GPT-5. P2 uses GPT-5.4-mini for the LLM-based schema matchers. P3 uses GPT-5.5 for schema matching and GPT-5.2 for labeling tasks. 1) Schema Matching: Table IV reports schema matching F1 per task. All three pipelines rely on the PyDI LLM schema matcher, with the human pipeline hand-correcting individual correspondences where needed. As reference, we add PyDI’s label-based and instance-based matchers. The human pipeline scores 100% on every task and the two automated pipelines stay within 2.9 points. The two
TABLE IV S CHEMA MATCHING F1 (%) ON THE M A DI-B ENCH BASE TASKS .
Task
TABLE V B LOCKING PAIR COMPLETENESS (PC), REDUCTION RATIO (RR), ENTITY MATCHING F1 (%).
Human BoB LLM-based Label-based Instance-based (P1) (P2) (P3) schema matching schema matching
Games 100.00 100.00 Companies 100.00 100.00 Music 100.00 100.00 Products 100.00 100.00 Papers 100.00 97.14
100.00 97.56 100.00 100.00 100.00
30.00 26.10 28.60 61.60 73.80
60.50 53.80 64.50 7.70 33.30
baselines show that the mapping is not trivial, and they fail in complementary places. Label-based matching peaks at 73.8% on Papers, where the source headers stay descriptive, and falls to 26.1% on Companies, where FullContact uses anonymized columns such as Attribute_1 (Table I). Instance-based matching shows the reverse profile: 64.5% on Music, where it infers the equally opaque MusicBrainz columns from their values, but 7.7% on Products, possibly because many of the 25 technical attributes, such as read and write speeds, draw values from similar numeric ranges. The Products headers, although encoded in unrelated conventions such as sequential_read_mb_s versus rd_mbs, stay descriptive enough for label-based matching to reach 61.6%, its second-best score. The benchmark thus stresses naming and value heterogeneity in different base tasks, and only methods that read names and values together, a human or an LLM, recover the mapping everywhere. 2) Blocking and Entity Matching: Table V reports blocking pair completeness and reduction ratio together with entity matching F1 per task, averaged over the source pairs. Pair completeness stays between 94.37% and 100.00% across all pipelines and tasks, close to or above the 97% floor that P1 and P2 tune for, so the candidate sets preserve almost all true matches and the entity matching results below reflect matcher quality without losing matching candidate pairs. The reduction ratios separate the pipelines only on Products: the human and best-of-breed pipelines prune 71.91% of the candidate space, which the smallest task tolerates, while the LLM-based pipeline’s embedding blocker removes 97.54% at 98.00% pair completeness. On every other task all pipelines exceed 99.2%, and on Papers, the largest task, at least 99.97%, where anything less would make the matching workload prohibitive. The size of the surviving candidate set also drives the runtime results discussed in Section III-C4. The Matching F1 columns of Table V report entity matching F1 on the labeled test sets, averaged over the source pairs of each task. The task scores span 63.09 to 99.70 F1, covering the range from near-solved to clearly open. On Companies, Music, and Papers the three pipelines are closely matched, within at most 3.7 points per task. Games and Products separate them. The human pipeline leads on Games with 89.45 against 67.30 and 63.15 for the automated pipelines. A possible explanation lies in the corner cases the test set is focused on: identical titles recur across platforms, and the rule that the same game on another platform is a different entity (Section II-B) penalizes matchers that rely on title similarity. The best-of-breed pipeline
PC (%) Task
P1
P2
RR (%) P3
P1
P2
Matching F1 (%) P3
P1
P2
P3
Games 99.55 100.00 96.85 99.93 99.90 99.96 89.45 67.30 63.15 Companies 97.73 94.37 100.00 99.34 99.24 99.67 87.65 89.29 90.69 Music 96.55 96.34 99.85 99.67 99.67 99.88 95.35 94.84 98.11 Products 100.00 100.00 98.00 71.91 71.91 97.54 70.37 84.09 63.09 Papers 97.90 97.90 97.69 99.98 99.98 99.97 98.45 99.70 96.05
leads on Products with 84.09 against 70.37 for the human pipeline, using the PLM-based Ditto matcher, which has been shown to work well for the products domain [21]. The LLMbased pipeline achieves the best scores on Companies (90.69) and Music (98.11). No method performs best on all tasks. The absolute values are in the range that current matchers report on established entity matching benchmarks such as WDC Products [15] and the bibliographic DBLP-ACM and DBLP-Scholar [21], so the matching difficulty is representative of real product and bibliographic data. As fusion builds on the output of this step, the lower matching performance on Games and Products will resurface in the fusion and end-to-end results (Section III-C4). 3) Data Fusion: Table VI reports fusion accuracy on the annotated holdout fusion records, scoring whether a pipeline produces the correct target value after matching and conflict resolution. Fusion is the most challenging step for every pipeline, with accuracies of 40.2 to 84.9, well below the schema matching and entity matching results. Two patterns stand out. First, the LLM-based pipeline reaches the highest accuracy on four of five tasks and exceeds the human configuration on all five, by 18.5 points on Companies (63.24 vs. 44.70) and 13.2 on Games (84.87 vs. 71.70): the LLMtuned conflict-resolution heuristics improve on the handchosen rules. Second, Products shows error propagation. Bestof-breed led entity matching there, and the two pipelines that trailed it, trail again in fusion, at 40.20% and 43.65% against its 56.68%, since a missed correspondence or a wrong match bounds what fusion can recover. An example is the Kingston DataTraveler 100 G3 flash drive from the human workflow: it appears in all four sources, but the matcher links only the source 1 record with a read speed of 100 MB/s, while the annotated value of 130 sits in an unmatched record. The missing record never enters the cluster, so no conflict-resolution rule can recover the correct value. Fusion is therefore the step with the most headroom on MaDI-Bench: the best score per task leaves between 15.1 (Games) and 43.3 (Products) points open. 4) End-to-End Results: Table VII reports the results of the end-to-end evaluation. The table separates end-to-end quality into reference-free, silver-reference, and ground-truth views for the two automated pipelines. The human pipeline provides the silver reference and is therefore not reported itself. Reference-free: Both pipelines expand the largest input source by roughly one third, with entity gains of 0.27 for bestof-breed and 0.32 for the LLM-based pipeline. The density
TABLE VI DATA FUSION ACCURACY (%) ON THE M A DI-B ENCH BASE TASKS .
Task
Human BoB LLM-based (P1) (P2) (P3)
Games Companies Music Products Papers
71.70 44.70 70.20 40.20 78.90
64.91 45.90 76.92 56.68 61.04
84.87 63.24 83.12 43.65 79.35
loss expected under this expansion, as many added entities are described by only one source, is nearly identical at −0.10 and −0.09. Therefore the larger output does not come at the cost of a much sparser table. The main structural difference is clustering: best-of-breed grows the entity count less but fuses more often, with a fusion ratio of 0.40 versus 0.33. The pertask spread of this ratio reflects both task structure and matcher behavior. On Products, where the four sources describe largely overlapping offers, 0.88 and 0.67 of output records merge several sources. On Games, where most entities appear in only one source, the ratio stays at 0.11 and 0.18. On a high-overlap task, a low ratio instead signals a matcher that leaves records as singletons. The reference-free view flags these structural differences but cannot judge whether the merges are correct. Schema validity is also close, with averages of 0.95 for best-of-breed and 0.94 for the LLM-based pipeline. High results such as the 1.00 in Games for best-of-breed are possible as strict checks cover dates, scores, ESRB labels, and list length, while the genre and platform taxonomies are intentionally non-exhaustive. A non-exhaustive taxonomy does not contain every possible value, so the validity check does not penalize values outside the taxonomy. Companies drops to 0.93 because industry, country, and key-people values violate target-schema constraints. The rows below quantify how much this appearance hides: the same outputs that reach 0.94 to 0.95 schema validity reach ground-truth fully-correct rates of 0.05 and 0.14. Reference-free metrics confirm target-format compatibility, and the reference-based rows are needed to evaluate factual quality. Silver reference: Against the human PyDI output, the LLM-based pipeline is closer to the human clustering structure. It reaches 0.57 entity recovery and 0.80 BCubed F1, compared with 0.52 and 0.75 for best-of-breed. The difference comes mainly from Companies and Music, where the LLMbased pipeline recovers more human clusters. Best-of-breed is closer on fused values: value drift is 0.16 instead of 0.20, fusion accuracy is 0.69 instead of 0.64, and the fully-correct rate is 0.21 instead of 0.06. The LLM-based pipeline more often recreates the human grouping, while best-of-breed more often selects the same values once an entity is aligned. Ground-truth reference: On the test sets, best-of-breed achieves the best clustering result. It recovers 0.90 of labeled entities on average compared with 0.81 for the LLM-based pipeline, especially on Products where the gap is 0.70 to 0.21, and BCubed F1 follows at 0.97 versus 0.94. The value-
level metrics show a different view of the results: Best-ofbreed has lower distribution drift, 0.48 versus 0.56, and the same value-density delta, −0.15, but the LLM-based pipeline has higher fusion accuracy, 0.71 versus 0.61. Matching the aggregate value distribution therefore does not imply assigning the correct values to the correct entities. The ground-truth reference also reverses the silver ranking on both axes. While the LLM-based pipeline performed best on clustering (BCubed F1 0.80 versus 0.75) and best-of-breed on values (fusion accuracy 0.69 versus 0.64) on the silver reference, the ground-truth reference shows the opposite in each case (0.94 versus 0.97 and 0.71 versus 0.61). This shows that agreement with a reference pipeline and correctness on independently annotated records show two different sides of the fused result. The strict fully-correct rate remains low for both pipelines: 0.05 for best-of-breed and 0.14 for the LLM-based pipeline. This metric requires every evaluated attribute of an entity to be correct at once, so it compounds the per-step errors: even though the matching correctness is high with a BCubed F1 of 0.94, only at most one in seven ground truth entities are fully correct, highlighting the difficulty of the fusion step. These metrics give a practitioner valuable insights into where they should look closer to debug and improve their integration system. Runtime: Table VIII reports wall-clock runtime in seconds for each of the three pipelines. All values measure the execution of the final pipeline. Model training is part of the measured runtime, while the configuration search and the human design effort are not. The pipelines differ by more than an order of magnitude, with mean runtimes of 234 seconds for the human configuration, 540 for the LLM-based pipeline, and 5,900 for best-of-breed. The two steps with the largest influence on runtime are blocking and entity matching. Papers, the largest task, is the most expensive task for every pipeline even at reduction ratios above 99.97% (Table V). Most of the bestof-breed entity matching runtime is the per-task fine-tuning and inference of Ditto. Companies, the only task on which the committee selects the cheaper Magellan, completes in 213 seconds, close to the LLM-based pipeline’s 203, while the Ditto-based tasks take 1,388 to 16,184 seconds. These costs put the effectiveness results into perspective: on Papers, bestof-breed spends roughly 25 times the runtime of the human configuration to gain 1.25 F1 in entity matching (Table V). D. Discussion The validation establishes four properties of the benchmark. First, no pipeline wins on every task: the human pipeline does best on entity matching on Games, best-of-breed for Products, and the LLM-based pipeline achieves the best fusion results on four of five tasks. Second, the end-to-end metrics expose additional weaknesses of each method: best-of-breed reaches 0.97 BCubed F1 under ground truth but only 0.61 fusion accuracy and a 0.05 fully-correct rate, and the silver and ground-truth metrics rank the two automated pipelines in
TABLE VII E ND - TO - END METRICS FOR THE BASE TASKS . ↑/↓ GIVE THE PREFERRED DIRECTION .
Ground Truth Reference
Silver Reference
Ref-Free
Dimension
Metric
Games
Companies
Music
Products
Papers
Mean
BoB (P2)
LLM-based (P3)
BoB (P2)
LLM-based (P3)
BoB (P2)
LLM-based (P3)
BoB (P2)
LLM-based (P3)
BoB (P2)
LLM-based (P3)
BoB (P2)
LLM-based (P3)
Entity Gain Density Gain ↑ Output Density ↑ Fusion Ratio
+0.40 −0.25 0.66 0.11
+0.32 −0.23 0.67 0.18
+0.19 −0.14 0.79 0.13
+0.27 −0.07 0.58 0.09
+0.38 −0.12 0.81 0.14
+0.37 −0.11 0.87 0.17
+0.13 +0.06 0.59 0.88
+0.22 +0.06 0.64 0.67
+0.24 −0.03 0.88 0.74
+0.41 −0.08 0.80 0.53
+0.27 −0.10 0.74 0.40
+0.32 −0.09 0.71 0.33
Consistency Schema validity ↑
1.00
0.85
0.93
0.99
0.84
0.97
1.00
1.00
0.98
0.88
0.95
0.94
Entity recovery ↑ Value drift ↓ Value-density ∆
0.50 0.10 −0.23
0.39 0.25 −0.24
0.37 0.35 −0.34
0.70 0.34 −0.22
0.76 0.29 −0.11
0.85 0.40 −0.08
0.04 0.04 −0.02
0.04 0.03 −0.03
0.93 0.01 −0.01
0.84 0.00 −0.05
0.52 0.16 −0.14
0.57 0.20 −0.12
Consistency Schema validity ∆ ↑ +0.00
−0.13
+0.06
−0.14
−0.10
−0.14
+0.00
+0.00
+0.17
−0.09
+0.03
−0.10
BCubed precision ↑ BCubed recall ↑ Correctness BCubed F1 ↑ Fusion acc. ↑ Fully-correct rate ↑
1.00 0.63 0.77 0.79 0.29
0.94 0.59 0.73 0.59 0.04
0.95 0.66 0.78 0.55 0.11
1.00 0.81 0.89 0.67 0.12
1.00 0.69 0.82 0.69 0.12
1.00 0.90 0.95 0.48 0.00
0.43 0.38 0.41 0.50 0.00
0.49 0.42 0.45 0.74 0.15
0.99 0.97 0.98 0.91 0.55
1.00 0.92 0.96 0.72 0.00
0.87 0.67 0.75 0.69 0.21
0.88 0.73 0.80 0.64 0.06
1.00 0.43 −0.20
1.00 0.67 −0.20
0.91 0.61 −0.22
0.88 0.66 −0.24
0.87 0.65 −0.03
0.96 0.71 +0.00
0.70 0.18 −0.24
0.21 0.25 −0.25
1.00 0.52 −0.07
1.00 0.53 −0.08
0.90 0.48 −0.15
0.81 0.56 −0.15
Consistency Schema validity ∆ ↑
0.00
−0.15
−0.07
−0.01
−0.16
−0.03
0.00
0.00
−0.02
−0.12
−0.05
−0.06
BCubed precision ↑ BCubed recall ↑ Correctness BCubed F1 ↑ Fusion acc. ↑ Fully-correct rate ↑
1.00 1.00 1.00 0.65 0.27
1.00 1.00 1.00 0.85 0.27
1.00 0.95 0.97 0.46 0.00
1.00 0.93 0.96 0.63 0.08
1.00 0.93 0.96 0.77 0.00
1.00 0.98 0.99 0.83 0.22
0.98 0.85 0.91 0.57 0.00
0.94 0.62 0.75 0.44 0.00
1.00 1.00 1.00 0.61 0.00
1.00 1.00 1.00 0.79 0.12
1.00 0.95 0.97 0.61 0.05
0.99 0.91 0.94 0.71 0.14
Coverage
Coverage
Coverage
Entity recovery ↑ Value drift ↓ Value-density ∆
TABLE VIII E ND - TO - END RUNTIME IN SECONDS FOR THE EVALUATED PIPELINES .
the value of such memorization. IV. VARIANT G ENERATION
Human (P1)
BoB (P2)
LLM-based (P3)
Games Companies Music Products Papers
180 70 184 98 637
5736 213 5980 1388 16184
196 203 211 718 1374
Mean
234
5900
540
Task
opposite order on both clustering and value correctness, the full picture only takes shape when looking at all metrics. Third, the benchmark offers ample room for improvement: fusion accuracy ranges from 40.2 to 84.9 and the ground-truth fullycorrect rate stays at or below 0.14. Fourth, when looking at the efficiency dimension, with mean runtimes of 234, 540, and 5,900 seconds, the effectiveness of a pipeline can be further weighed by its efficiency. Two limitations apply. First, the silver reference is produced by the human PyDI pipeline, so silver metrics may favor systems whose decisions resemble PyDI-style workflows. Depending on the silver standard a practitioners chooses, it is important to keep potential biases in mind. The hand-annotated ground-truth sets are independent of one specific pipeline and give a stronger correctness estimate, although they only cover a small subset of the overall output dataset. Second, the source data is partly public, so LLM-based systems may have seen parts of it during pretraining. The perturbed task variants (Section IV) and the unpublished fused target tables reduce
The five base tasks employ real-world datasets and have a fixed level of difficulty. Given the current pace of improvement in the area of agentic end-to-end data integration systems, it is likely that the benchmark tasks get saturated quickly, making them unsuitable for evaluation of data integration systems in the mid-term. On the other hand, the current difficulty of some of the tasks may already be higher than what a practitioner may face on tasks in their domain, making the application of simpler less costly methods unsuitable for the benchmark, while they may prove to be effective and efficient alternatives for the use cases faced by the practitioners. In order to address both of these cases, we derive three variants of each base task, namely easy, medium, and hard. The variants hold the underlying integration problem fixed but vary the level of difficulty of the subtasks. Because the variants amplify or reduce the same forms of heterogeneity that make real integration hard, a harder or easier level corresponds to genuinely more or less demanding real-world conditions. As a result, the benchmark may stay future-proof as systems improve, while also supporting the evaluation in less demanding scenarios where efficiency is the main concern. Each base task is released together with its three variants, so MaDI-Bench offers 20 integration tasks across five domains in total. At a high level, the variants differ from the base tasks in how hard the same integration problem is to solve. Across the easy, medium, and hard levels, we reduce surface overlap by injecting value noise such as typos and encoding artefacts,
TABLE IX A DB PEDIA C OMPANIES RECORD AT BASE AND hard DIFFICULTY. C OLORS MARK THE RESPONSIBLE KNOB : SCHEMA NAMING DIVERGENCE (7), SURFACE AUGMENTATION (1), FORMAT AND UNIT DIVERSITY (5), VALUE NOISE (6), AND ATTRIBUTE DROP (3). Attribute (base → hard)
Base value
Hard value
org_name → nm established → ey nation → cn sector → sg keypeople_name → kpn total_assets_val → ta annual_income → ai
Taisei Corporation 1873-01-01 Japan Construction Okura Kihachiro 3650187000 213195000
Taisei Corp 01.01.187312:00:00 Nippon Constructi on dropped 3650.19 167.18
add negative and positive corner-case records to the clusters describing specific entities, drop cell values and skew which sources cover each entity, diversify date, number, and unit formats, rename source columns to synonyms or opaque codes, and shuffle which source holds the correct fusion value so that no single source is more trustworthy in general. Importantly, these changes never alter the correspondences among the underlying records, a match on easy does not become a nonmatch on hard due to value perturbation, only the difficulty is adjusted so the decision moves further or closer to the boundary between matches and non-matches. As a result, a single dataset spans a controlled difficulty range on which the increasing difficulty impacts all steps of the integration workflow. The augmentation method that derives the variants perturbs the source data along a set of independently controllable difficulty knobs, each set to an absolute target for the three difficulty levels, for example, a corner-case ratio of 20/50/80% for easy/medium/hard or a per attribute drop rate of 5/15/35%, and each targeting a specific challenge of the integration task. As a result, a variant can add or remove heterogeneity relative to the domain’s base task (Section IV-A). The knobs are applied in a fixed order so that the perturbations compose predictably, while some entities and values are protected to ensure the evaluation sets remain useable (Section IV-B). After creation, a validation step confirms that the resulting variants are progressively harder by running heterogeneous algorithm committees for each pipeline stage and reporting the committee mean and ceiling across difficulty levels and against the base task (Section IV-C). Finally, Section IV-D reports variant statistics together with best-of-breed and LLMbased pipeline results on the variants relative to the base tasks. A. Difficulty Knobs We control the difficulty of an end-to-end DI task using eight knobs. Each knob is an independently togglable perturbation that targets one or more difficulty dimensions of a specific pipeline step, with absolute targets per easy/medium/hard level. We describe the eight knobs in turn, naming in parentheses the difficulty dimension that each effect targets. Table IX illustrates the value-level knobs on a single Companies record at hard difficulty.
1) Surface augmentation intensity rewrites attribute values into alternative surface forms using abbreviations, token reorderings, reformulations, and dropped tokens, eroding the surface overlap that blocking and matching rely on (representation heterogeneity, corner-case difficulty) and yielding plausible value disagreements for fusion (conflict subtlety). The levels rewrite 0/16/30% of values. 2) Entity niche density adds similar-but-distinct entities within the same domain niche, such as many products of one brand, enlarging blocks with non-matching pairs (candidate density) and pushing candidate pairs toward the matching decision boundary (corner-case ratio). The corner-case ratio is set to 20/50/80%. 3) Per-source attribute drop rate blanks 5/15/35% of cells per source, removing blocking keys (blocking-key completeness), matching evidence (record completeness), and the redundancy that fusion cross-validates against (source density). 4) Per-entity coverage skew varies how many sources describe each entity. The share of single-source entities rises over 0/20/55%, skewing group sizes and thinning the evidence for fusion (source density, conflict rate). 5) Format and unit diversity rewrites dates, numbers, and quantities into two to four admissible formats per type and one to three locales, mixing units and scales at harder levels (format heterogeneity, unit and scale diversity). 6) Value-noise injection rate corrupts 1/4/12% of values with typos, encoding artefacts, and truncation from the FEBRL corruption taxonomy [30], degrading similarity signals for matching and surfacing as source disagreements for fusion (noise and corruption, corner-case difficulty, conflict rate). 7) Schema naming divergence renames source columns from fully descriptive (easy) through abbreviations (medium) to opaque codes (hard), removing the attribute name signal that schema matchers rely on (naming heterogeneity). 8) Source reliability differentiation reshuffles which source holds the correct value per attribute. The most reliable source holds the correct value for 85/65/50% of an attribute’s entities, so that no fixed source-trust strategy stays optimal (trust ambiguity, conflict rate). B. Generation Method The following paragraphs describe the relevant artefacts used in the creation of the variants. Positive Pool: The variant generator must be able to perturb the data without destroying the existing cross-source matches, i.e. it should not change a match into a non-match. The labeled entity matching training, validation and test sets cover only a small sample of all true pairs.Therefore, we build a per-domain positive pool: an approximation of the real matches, assembled by merging the labeled positives, the correspondences emitted by each domain’s human-built baseline matcher, and the predictions of a domain-trained Ditto [21] matcher over a blocker-derived candidate set. Pairs on which the baseline matcher and Ditto agree enter the pool directly. The remaining disagreements are arbitrated by an LLM (GPT-5.4), and the pool is closed transitively across source pairs so that a match
inferred for one pair propagates to the others. For the products and papers domains, we have full linkage information due to the existing cluster id and DOI attributes, and build the positive pool directly. Protected Entities and Values: The entities in the fusion validation and test sets form a protection set, whose members are prohibited from being dropped during variant generation. Entity niche density (Knob 2) is not allowed to touch any records pertaining to these entities. Consequently, every entity referenced by the fusion validation or test set survives at every difficulty level by construction, so the fusion validation and test sets remain intact for evaluation. A strong perturbation could still corrupt every source value of a fused cell beyond recovery. To handle this, for each fusion value, the generator enforces a closeness floor: at least one surviving source value for an entity must stay within an attributespecific tolerance of the target value (±3% for continuous quantities, ±1 year for dates, and normalized Levenshtein or token-Jaccard similarity thresholds of 0.5–0.85 for strings), so that a suitable fusion strategy can still recover the correct value. The value-corruption knobs (surface augmentation and value noise) mutations are discarded whenever they would push the last source value of an entity out of tolerance. Format and unit changes (Knob 5) are exempt, as they are equivalence transforms that can be reverted to the same fused value. The target values used for protection come from the fusion validation and test sets. To extend protection beyond those two hundred annotated entities for each domain to the full set of entities, we also use a fusion silver standard. This silver standard consists of the fused values for each entity from each domain’s hand-built fusion pipeline (P1). This extends the fusion value protection to the full tasks. Evaluation Sets: The fusion validation and test sets are copied unchanged across all difficulty levels. The entity matching evaluation sets are copied from the base tasks. Any record pairs that no longer exist due to dropped records are removed and subsequently filled with positive and negative corner-cases until they have a similar size and positive ratio as the base task (Section IV-C). The schema matching test set is rewritten to fit the Knob 7 renames. C. Variant Validation and Statistics To measure the difficulty of the created variants, we measure the impact of the changes on each step of the integration pipeline in comparison to the base tasks. For each stage of the pipeline, we run a committee of algorithms (Table X) on every difficulty level. Every member is fitted once on the base task and then scored unchanged on each variant. Each committee reports two measurements: the mean, the macro-average over all members, and the ceiling, the score of the strongest member at each level of difficulty. Raising the difficulty should depress both, so that a variant challenges not only the average member but also the best method. Table XI reports both metrics per stage, alongside the variant statistics. The statistics show that all variants keep close to the row
TABLE X C OMMITTEE MEMBERS PER PIPELINE STAGE . Stage
Committee members
Schema matching (7)
label-based, instance-based and duplicate-based matching, Sentence-BERT [18], COMA [16], Magneto [17], GPT-5.4-mini standard, token, sorted-neighborhood [4], Sentence-BERT [18], BM25 [19], SC-Block [20] Magellan [22], Ditto [21], MatchGPT [31], ComEM [32] Voting, source-trust-based, PyDI per-attribute, TruthFinder [23], AccuSim [25], LTM [24], FusionQuery [27], CASEFusion [26]
Blocking (6) Entity matching (4) Data fusion (8)
TABLE XI VARIANT STATISTICS & COMMITTEE DIFFICULTY SIGNAL ACROSS LEVELS . Committee mean (%)
Sources Domain
Lvl
Rows (k) Density
Companies
Base Easy Med. Hard
14.0 14.8 14.3 14.2
Games
Base Easy Med. Hard
Music
Best-member ceiling (%)
SM
EM
Fus.
SM
EM
Fus.
66 60 59 58
72.7 86.2 77.0 64.6
88.4 89.9 87.2 86.8
40.4 35.8 38.2 32.4
100.0 100.0 100.0 100.0
89.5 93.4 88.2 88.1
45.8 40.0 41.3 34.8
75.0 78.9 75.3 71.1
94 84 83 83
74.8 87.1 77.0 70.9
60.9 63.3 64.8 55.3
68.2 100.0 67.1 98.1 65.0 98.1 55.8 98.1
71.6 71.8 75.5 71.5
72.0 71.1 68.7 58.6
Base Easy Med. Hard
37.3 39.7 37.3 35.2
92 75 70 69
74.3 90.0 76.8 65.5
82.4 88.3 82.7 79.2
75.5 100.0 83.6 100.0 69.9 100.0 54.5 89.8
95.3 95.1 94.8 92.9
77.9 86.6 76.5 56.6
Products
Base Easy Med. Hard
3.0 3.1 3.1 2.4
56 48 44 48
67.6 74.5 68.6 62.1
90.0 91.9 90.0 85.7
59.1 44.7 51.8 45.5
98.1 99.5 98.1 98.1
93.4 95.9 97.9 93.4
69.8 54.1 61.1 51.4
Papers
Base Easy Med. Hard
182.1 201.5 201.5 149.6
74 66 63 60
83.4 75.9 58.2 49.4
96.6 96.5 95.3 92.2
54.8 52.7 42.8 38.8
97.1 92.9 90.5 92.9
100.0 99.9 99.9 99.4
61.0 58.3 47.7 43.5
counts of the base sources, with density decreasing from easy to hard. The easy committee mean exceeds the base score for schema and entity matching in four of five domains, confirming that the easy level relaxes the integration problem below the level of difficulty of the base task. Fusion is the exception at easy, as attribute dropping and value noising also happen at this level. From medium to hard, both the mean and the ceiling decline at almost every step. The signal is clearest for fusion, whose mean and ceiling fall from base to hard in every domain. The committees validate the variants as a monotone difficulty ladder with the main burden falling on fusion. For schema matching the mean and the ceiling diverge: the mean drops under the header renames of Knob 7, which collapse the label-focused matchers, while the strongest instanceand LLM-based matchers still recover the mapping from the values. The entity matching mean declines in most domains from easy to hard, but the best member (Ditto) is affected to a lesser degree.
TABLE XII P ER - STAGE SCORES OF B O B (P2) AND THE LLM- BASED PIPELINE (P3) ON THE BASE TASK AND ITS VARIANTS . BoB (P2) Domain
Lvl
SM
LLM-based (P3)
EM Fus.
SM
Base 100.0 89.3 45.9 97.6 Easy 100.0 93.6 40.3 100.0 Companies Med. 100.0 88.9 39.6 100.0 Hard 91.3 94.5 33.3 100.0
EM Fus. 90.7 88.2 77.0 87.4
63.2 42.3 56.8 44.4
Games
Base 100.0 67.3 64.9 100.0 63.2 84.9 Easy 98.1 66.9 62.1 97.9 76.0 81.4 Med. 98.1 69.7 63.6 97.9 72.7 78.5 Hard 98.1 79.2 52.5 97.9 82.0 72.6
Music
Base 100.0 94.8 76.9 100.0 98.1 83.1 Easy 100.0 94.9 85.4 100.0 97.7 77.4 Med. 100.0 93.2 65.1 100.0 90.9 71.0 Hard 92.0 94.3 54.5 90.9 88.0 58.6
Products
Base 100.0 84.1 56.7 100.0 63.1 43.6 Easy 99.5 91.5 51.6 100.0 27.5 49.3 Med. 98.1 88.1 59.3 100.0 36.9 45.9 Hard 98.1 85.3 49.5 98.0 51.0 34.1
Papers
Base Easy Med. Hard
97.1 92.9 90.5 92.9
99.7 99.5 99.2 96.7
TABLE XIII E VALUATION SCOPE OF RELATED BENCHMARKS : EFFICIENCY (E FF .), END - TO - END OUTPUT QUALITY (E2E), AND STEP - LEVEL EVALUATION OF SCHEMA MATCHING (SM), BLOCKING (B L .), ENTITY MATCHING (EM), AND DATA FUSION (F US .). ✓: YES , ◦: PARTIAL , ×: NO .
61.0 100.0 96.0 79.3 58.1 91.7 90.9 64.6 47.7 91.7 87.6 62.3 37.8 91.7 78.1 51.1
D. Variant Evaluation Table XII reports the per-stage scores of BoB (P2) and the LLM-based pipeline (P3) on each base task and its three variants. Compared against the base task column, the variants produce the intended difficulty gradient, most clearly for data fusion. For best-of-breed the hard variant gives the lowest fusion score in every domain, with base-to-hard losses of −12.6 (Companies), −12.4 (Games), −22.4 (Music), −7.2 (Products), and −23.2 (Papers). The LLM-based pipeline behaves the same way, with base-to-hard losses of up to −28.2 (Papers) and −24.5 (Music). Schema matching is the most robust stage. The LLM and instance-based matchers recover the mapping from the column values. Entity matching is similarly resilient once the matcher is re-tuned per variant. The two automated pipelines respond to difficulty in opposite ways at the two hardest stages. Best-of-breed is more robust in entity matching: its Ditto matcher holds 84.1–91.5% on Products across all levels and never falls below 96.7 on Papers, whereas the LLM-based pipeline collapses on Products and declines steadily on Papers. This is expected, as PLMbased matchers handle heterogeneity better than similarityfeature-based matchers [21]. The picture reverses for fusion: the LLM-based pipeline attains the higher accuracy at every level on Companies, Games, and Papers, and on most levels of Music. The exception is Products, where best-of-breed scores higher throughout, a likely consequence of less error propagation from its stronger entity matching on that domain. In summary, the hard variants raise the overall integration difficulty in every domain, preserving headroom for future systems to improve, whereas the easy variants support the
Benchmark
Eval. unit
Eff. E2E SM Bl. EM Fus.
TPC-DI [13] DIPBench [33] DIBS [34] Alaska [35] LakeBench [36] ELT-Bench [14] KGI-Bench [37] MaDI-Bench
warehouse load process execution preprocessing kernels schema matching/ER join/union discovery agent ELT pipeline knowledge graph fused table
✓ ✓ ✓ × ✓ ✓ ✓ ✓
× × × × × ✓ ✓ ✓
× × × ✓ × × ✓ ✓
× × × × × × ◦ ✓
× × × ✓ × × ✓ ✓
× × × × × × ✓ ✓
evaluation of simpler and computationally cheaper methods on less demanding integration problems. V. R ELATED W ORK This section compares MaDI-Bench to existing benchmarks in data integration and data engineering. TPC-DI, DIPBench, and DIBS measure the efficiency of integration systems. TPC-DI models a retail-brokerage ETL workload with fixed source and target models, transformations, run rules, and a throughput metric, so it evaluates how efficiently a system executes the specified load rather than whether it can infer integration decisions [13]. DIPBench defines source and target schemas, 15 integration process types, concurrent execution streams, and scale factors for measuring integration-process execution [33], while DIBS provides preprocessing kernels for parsing, transformation, and aggregation and characterizes their hardware behavior [34]. Alaska exposes schema matching and entity resolution tasks over about 70k Web product specifications with manually curated ground truth [35]. LakeBench evaluates the effectiveness and efficiency of top-k discovery of joinable and unionable tables over more than 16 million tables and more than 10,000 labeled queries [36], addressing discovery before integration. ELT-Bench and KGI-Bench evaluate complete pipelines but target different artefacts. ELT-Bench evaluates whether agents can configure Airbyte/Snowflake pipelines and write dbt transformations for target data models, reporting stage success, target-model correctness, cost, and steps [14]. KGIBench evaluates pipelines for integrating data into knowledge graphs [37]. Similar to MaDI-Bench, KGI-Bench also covers all steps of the data integration process, but provides only a single integration task, integrating data describing movies into the DBpedia knowledge graph. A main difference between the benchmarks is that MaDI-Bench addresses the integration of relational tables, while KGI-Bench addresses the augmentation of RDF knowledge graphs. Table XIII compares the different benchmarks. The efficiency-oriented benchmarks score none of the integration decisions. Step-level evaluation is limited to Alaska’s schema matching and entity resolution ground truth and KGI-Bench for the task of augmenting knowledge graphs. The comparison
shows that MaDI-Bench is the only benchmark covering all steps of the integration process for the use case of integrating tabular data. VI. C ONCLUSION This paper introduced the Mannheim Data Integration Benchmark (MaDI-Bench). The benchmark models the full complexity of the data integration process, including schema matching, value normalization, entity matching, and conflict resolution, while at the same time taking into account the dependencies between the subtasks. To prevent a quick saturation of the benchmark as agentic systems progress, we introduce a generic variant-generation method for deriving harder as well as easier variants from the base tasks. We validated MaDI-Bench and the generated variants using humandesigned pipelines, an LLM-based pipeline, and a best-ofbreed pipeline. The validation showed the benchmarks utility for measuring the step-wise as well as end-to-end performance of data integration pipelines. We hope that the benchmark will prove useful for the community and will support the development of fully automatic as well as human in the loop data integration systems. ACKNOWLEDGMENT OF AI-G ENERATED C ONTENT During the preparation of this work, the authors used Claude Code (Anthropic) and OpenAI Codex to assist with (i) developing and testing the benchmark code and evaluation scripts and (ii) refining the manuscript text across all sections. The authors reviewed and verified all AI-assisted content and take full responsibility for the contents of this publication. The design of the benchmark, its motivation, as well as the interpretation of the experimental results are exclusively the authors’ own. R EFERENCES [1] A. Doan, A. Halevy, and Z. Ives, Principles of Data Integration. Morgan Kaufmann, 2012. [2] X. L. Dong and D. Srivastava, Big data integration. Morgan & Claypool Publishers, 2015. [3] P. A. Bernstein, J. Madhavan, and E. Rahm, “Generic Schema Matching, Ten Years Later,” Proceedings of the VLDB Endowment, vol. 4, no. 11, pp. 695–701, 2011. [4] V. Christophides, V. Efthymiou, T. Palpanas et al., “An Overview of End-to-End Entity Resolution for Big Data,” ACM Computing Surveys, vol. 53, no. 6, pp. 1–42, 2020. [5] J. Bleiholder and F. Naumann, “Data Fusion,” ACM Computing Surveys, vol. 41, no. 1, pp. 1–41, 2009. [6] A. Steiner and C. Bizer, “Automatic End-to-End Data Integration using Large Language Models,” 2026. [7] Y. Zhu, L. Wang, C. Yang et al., “A Survey of Data Agents: Emerging Paradigm or Overstated Hype?” arXiv preprint arXiv:2510.23587, 2025. [8] X. Zhou, J. He, W. Zhou et al., “A Survey of LLM × DATA,” arXiv preprint arXiv:2505.18458, 2025. [9] K. Chen, P. Wang, Y. Yu et al., “Large Language Model-based Data Science Agent: A Survey,” arXiv preprint arXiv:2508.02744, 2025. [10] C. Koutras, G. Siachamis, A. Ionescu, K. Psarakis, J. Brons, M. Fragkoulis, C. Lofi, A. Bonifati, and A. Katsifodimos, “Valentine: Evaluating matching techniques for dataset discovery,” in 2021 IEEE 37th International Conference on Data Engineering (ICDE). IEEE, 2021, pp. 468–479. [11] S. Mudgal, H. Li, T. Rekatsinas, A. Doan, Y. Park, G. Krishnan, R. Deep, E. Arcaute, and V. Raghavendra, “Deep learning for entity matching: A design space exploration,” in Proceedings of the 2018 international conference on management of data, 2018, pp. 19–34.
[12] X. Yin, J. Han, and P. S. Yu, “Truth discovery with multiple conflicting information providers on the web,” in Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining, 2007, pp. 1048–1052. [13] M. Poess, T. Rabl, H.-A. Jacobsen et al., “TPC-DI: The First Industry Benchmark for Data Integration,” Proceedings of the VLDB Endowment, vol. 7, no. 13, pp. 1367–1378, 2014. [14] T. Jin, Y. Zhu, and D. Kang, “ELT-Bench: An End-to-End Benchmark for Evaluating AI Agents on ELT Pipelines,” Proceedings of the VLDB Endowment, vol. 19, no. 2, pp. 84–98, 2025. [15] R. Peeters, R. C. Der, and C. Bizer, “WDC Products: A MultiDimensional Entity Matching Benchmark,” in Proceedings of the 27th International Conference on Extending Database Technology, 2024, pp. 22–33. [16] H.-H. Do and E. Rahm, “COMA: A System for Flexible Combination of Schema Matching Approaches,” in Proceedings of the 28th International Conference on Very Large Data Bases, 2002, pp. 610–621. [17] Y. Liu, E. H. M. Pena, A. Santos et al., “Magneto: Combining Small and Large Language Models for Schema Matching,” Proceedings of the VLDB Endowment, vol. 18, no. 8, pp. 2681–2694, 2025. [18] N. Reimers and I. Gurevych, “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, 2019, pp. 3982–3992. [19] S. Robertson and H. Zaragoza, “The Probabilistic Relevance Framework: BM25 and Beyond,” Foundations and Trends in Information Retrieval, vol. 3, no. 4, pp. 333–389, 2009. [20] A. Brinkmann, R. Shraga, and C. Bizer, “SC-Block: Supervised Contrastive Blocking Within Entity Resolution Pipelines,” in The Semantic Web: 21st International Conference, ESWC, 2024, pp. 121–142. [21] Y. Li, J. Li, Y. Suhara et al., “Deep Entity Matching with Pre-Trained Language Models,” Proceedings of the VLDB Endowment, vol. 14, no. 1, pp. 50–60, 2020. [22] P. Konda, S. Das, P. Suganthan G. C. et al., “Magellan: Toward Building Entity Matching Management Systems,” Proceedings of the VLDB Endowment, vol. 9, no. 12, pp. 1197–1208, 2016. [23] X. Yin, J. Han, and P. S. Yu, “Truth Discovery with Multiple Conflicting Information Providers on the Web,” in Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2007, pp. 1048–1052. [24] B. Zhao, B. I. P. Rubinstein, J. Gemmell et al., “A Bayesian Approach to Discovering Truth from Conflicting Sources for Data Integration,” Proceedings of the VLDB Endowment, vol. 5, no. 6, pp. 550–561, 2012. [25] X. L. Dong, L. Berti-Equille, and D. Srivastava, “Integrating Conflicting Data: The Role of Source Dependence,” Proceedings of the VLDB Endowment, vol. 2, no. 1, pp. 550–561, 2009. [26] S. Lyu, W. Ouyang, H. Shen et al., “Truth Discovery by Claim and Source Embedding,” in Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, 2017, pp. 2183–2186. [27] J. Zhu, Y. Mao, L. Chen et al., “FusionQuery: On-demand Fusion Queries over Multi-source Heterogeneous Data,” Proceedings of the VLDB Endowment, vol. 17, no. 6, pp. 1337–1349, 2024. [28] J. Lin, “Divergence Measures Based on the Shannon Entropy,” IEEE Transactions on Information Theory, vol. 37, no. 1, pp. 145–151, 1991. [29] A. Ramdas, N. Garcı́a Trillos, and M. Cuturi, “On Wasserstein TwoSample Testing and Related Families of Nonparametric Tests,” Entropy, vol. 19, no. 2, p. 47, 2017. [30] P. Christen, “FEBRL: An Open Source Data Cleaning, Deduplication and Record Linkage System with a Graphical User Interface,” in Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2008, pp. 1065–1068. [31] R. Peeters, A. Steiner, and C. Bizer, “Entity Matching using Large Language Models,” in Proceedings of the 28th International Conference on Extending Database Technology, 2025, pp. 529–541. [32] T. Wang, X. Chen, H. Lin et al., “Match, Compare, or Select? An Investigation of Large Language Models for Entity Matching,” in Proceedings of the 31st International Conference on Computational Linguistics, 2025, pp. 96–109. [33] M. Böhm, D. Habich, W. Lehner et al., “DIPBench Toolsuite: A Framework for Benchmarking Integration Systems,” in Proceedings of the 2008 IEEE 24th International Conference on Data Engineering, 2008, pp. 1596–1599.
[34] A. M. Cabrera, C. J. Faber, K. Cepeda et al., “DIBS: A Data Integration Benchmark Suite,” in Companion of the 2018 ACM/SPEC International Conference on Performance Engineering, 2018, pp. 25–28. [35] V. Crescenzi, A. De Angelis, D. Firmani et al., “Alaska: A Flexible Benchmark for Data Integration Tasks,” arXiv preprint arXiv:2101.11259, 2021.
[36] Y. Deng, C. Chai, L. Cao et al., “LakeBench: A Benchmark for Discovering Joinable and Unionable Tables in Data Lakes,” Proceedings of the VLDB Endowment, vol. 17, no. 8, pp. 1925–1938, 2024. [37] M. Hofer and E. Rahm, “Evaluation of Pipelines for Data Integration into Knowledge Graphs,” arXiv preprint arXiv:2605.22304, 2026.