Evaluation of Pipelines for Data Integration into Knowledge Graphs Marvin Hofer
ScaDS.AI Dresden/Leipzig Leipzig University, Germany [email protected]
arXiv:2605.22304v1 [cs.AI] 21 May 2026
ABSTRACT Integrating new data into knowledge graphs (KG) typically involves different tasks that are executed within workflows or pipelines There are many possible pipelines for a specific integration problem but there is not yet a general approach to evaluate the overall quality and performance of such pipelines to be able to determine the best choices. We therefore propose a new benchmark KGI-Bench to evaluate integration pipelines that ingest different kinds of input data into an existing KG. We evaluate pipelines by analyzing their output, i.e., the updated KG, with the three complementary quality metrics coverage, correctness and consistency. We also provide benchmark datasets (seed KG, overlapping input data of three formats, reference KG as a ground truth) for the movie domain. To demonstrate the applicability and usefulness of the proposed benchmark, we comparatively evaluate 12 pipelines and analyze their behavior across different input data formats and design choices. Artifact Availability: All code and datasets are available at: Repository: https://github.com/ScaDS/KGI-Bench Dataset: https://doi.org/10.5281/zenodo.17246357
1
INTRODUCTION
Knowledge graphs (KGs) have become essential for integrating and representing heterogeneous data in a unified and semantically rich form. Constructing and continuously updating KGs is a complex task involving solutions for multiple tasks, such as information extraction, data cleaning, ontology alignment, entity matching and entity fusion. A huge amount of research and development has led to powerful tools for these individual tasks that can also benefit from utilizing Large Language Models (LLMs) [13, 32]. The task approaches are typically developed independently and tailored to specific conditions (e.g., data formats) making it difficult to assemble them into reliable and reusable pipelines to construct and update KGs [17]. Currently, KG pipelines are mostly handcrafted for specific domains or tasks using ad hoc scripts and configurations. There are a few tools to develop pipelines for constructing and updating KGs but these are mostly not openly available or restricted to a single application and unable to reuse existing solutions for certain integration tasks [17]. A main open challenge lies in evaluating entire KG integration pipelines to identify the quality and problems of such pipelines and to compare the quality and efficiency of different pipelines with each other to help find the best integration approaches. This is challenging since pipelines consist of multiple complex tasks that are already difficult to evaluate. Furthermore the evaluation should
Erhard Rahm
ScaDS.AI Dresden/Leipzig Leipzig University, Germany [email protected] be possible for a large spectrum of possible pipelines to enable the integration of diverse data of different formats and to make use of different implementations for specific tasks including the use of LLMs. Furthermore, it has to be considered that pipelines typically update an already existing version of a KG in an incremental manner so that input data needs to be matched with each other and the KG to avoid that redundant or duplicate data is introduced in the KG. To address this gap, we propose a new benchmark approach called KGI-Bench (Knowledge Graph Integration Benchmark), to systematically assess end-to-end KG integration pipelines. It focuses on the evaluation of the result of a KG pipeline, i.e., the newly generated version of a KG after integrating one or several new input data sources. KGI-Bench uses a set of new evaluation metrics to determine the coverage, correctness and consistency of the updated KG. We also provide a domain-specific instantiation of the benchmark for the movie domain as a basis for evaluating concrete pipelines with a seed KG, overlapping input data of three data formats (text, JSON , RDF) as well as a reference graph as a ground truth for the quality evaluation. All datasets are made publicly available. Our main contributions are as follows: (1) We propose a general framework for evaluating KG integration pipelines, KGI-Bench, including quality metrics for coverage (to what degree is the input data covered in the updated KG), correctness (to what degree is added or changed data in the KG correct) and consistency of the updated KG. We also consider additional metrics, e.g., regarding pipeline efficiency. Some of the quality metrics are also applicable when no reference graph is available. We further support the weighted combination of the metrics into a single number to facilitate a ranking among several pipelines. (2) While KGI-Bench is generic we provide open benchmark datasets for non-trivial data integration for the movie domain (KGI-Bench-Movie). This enables the comparative evaluation of specific pipelines or pipeline tools. The provided benchmark datasets include a seed KG, overlapping input sources in RDF, JSON, and text, as well as a reference KG as a ground truth for the updated KG that includes all input data without duplicates. (3) To demonstrate the applicability and usefulness of KGI-Bench, we comparatively evaluate 12 integration pipelines (generated by the tool KGpipe [18]), some of which make use of a LLM. The remainder of the paper is structured as follows. After reviewing related work, we provide definitions and discuss requirements for end-to-end evaluation of KG integration pipelines. We then introduce the evaluation metrics of KGI-Bench. Section 5 presents
Marvin Hofer and Erhard Rahm
the benchmark datasets for the movie domain. Subsequently, we describe the evaluated pipelines and their comparative evaluation using KGI-Bench. Finally, we conclude and discuss future directions.
2
BACKGROUND AND RELATED WORK
There are several surveys on the approaches for constructing knowledge graphs covering the individual tasks (e.g., data acquisition, ontology design, knowledge extraction, entity resolution, entity fusion, knowledge completion,) as well as tools to define and execute integration pipelines [17, 19, 44, 47]. We will focus here on previous approaches for evaluating data integration for knowledge graphs. Most of the previous approaches focus on evaluating individual tasks but there also some approaches to evaluate the overall quality and usefulness of knowledge graphs. Benchmark datasets play a crucial role in both perspectives by providing common reference points for reproducible comparisons [15, 46, 48]. Task-level evaluation typically relies on ground truth datasets tailored to specific integration tasks. For entity resolution, evaluation typically uses gold standards of duplicate records or clusters, with metrics such as recall, precision and F-measure [8, 36]. Ontology and schema matching tasks rely on curated reference alignments and are also commonly evaluated using recall, precision, and F-measure as established in OAEI evaluation campaigns providing different benchmarks [12, 16]. Recent frameworks such as OntoAligner provide modular support for ontology alignment experiments across multiple benchmark tracks [14]. For schema matching across tabular datasets, evaluation is sometimes rankingbased, using metrics such as Recall@k to measure the retrieval of semantically related columns. The Valentine framework supports large-scale experiments for such scenarios [26]. Entity alignment across knowledge graphs uses datasets with known cross-graph correspondences, such as OpenEA, enabling evaluation of alignment methods using metrics such as Hits@k and MRR [43]. For relation extraction, annotated text corpora are used with micro-precision, micro-recall, and micro-F1 metrics, as illustrated by multilingual benchmarks such as RED𝐹 𝑀 [22]. Recent benchmarks also support text-to-KG extraction and ontology-driven generation tasks, such as Text2KGBench [33] and REDFM [22]. Evaluation datasets and training corpora for extraction, linking, and alignment tasks are often derived from large knowledge bases such as DBpedia and Wikidata [6]. There are different approaches for evaluating entire KGs but most of them do not consider the effect of specific integration pipelines. Several evaluation dimensions have been proposed to determine the quality of a KG such as correctness, coverage, consistency, timeliness, and usefulness for downstream applications [20, 37, 46, 48]. Recent work emphasizes that KG quality depends on interactions between data, sources, systems, and usage context [34]. Recent work has also investigated the evaluation of evolving knowledge graphs and ontologies. Bakker and de Boer propose syntactic and semantic metrics to automatically evaluate changes in knowledge graphs over time, focusing on structural and conceptual quality aspects such as hierarchy structure and semantic consistency between related concepts [3]. Their work mainly targets ontology evolution and schema-level change evaluation, while our work focuses on
benchmarking end-to-end KG integration pipelines and evaluating the resulting integrated KG with respect to source coverage, factual correctness, and ontology consistency. Because manual verification of all KG elements (e.g., RDF triples) is infeasible for large graphs, sampling-based approaches are often used to estimate accuracy (correctness) [29]. Other methods reduce annotation effort by propagating correctness judgments through logical constraints, as in KGEval [35]. In settings without a gold standard, comparative approaches such as ABECTO[23] estimate accuracy and completeness by exploiting overlap between existing knowledge graphs. Huaman et al. [21] proposed a KG validation approach based on matching entities across weighted external knowledge sources and computing confidence scores from feature similarity comparisons, and FactCheck [41] is a benchmark for LLM-based KG fact validation that evaluates internal model knowledge, retrieval-augmented verification, and multi-model consensus strategies on manually annotated KG triples. The quality and usefulness of a KG can also be evaluated extrinsically through downstream tasks such as classification or recommendation. KGrEaT exemplifies this approach by measuring how different KGs influence task performance [15]. Additional validation approaches include fact-checking frameworks that verify KG assertions using graph structure or external text corpora [1, 38]. Logical consistency is another important quality dimension. Constraint languages such as SHACL [9], ShEx [39] and RDFUnit [24] allow detection of violations of ontology constraints including domain, range, and cardinality restrictions. Sieve combines quality assessment and data fusion by attaching quality scores such as recency or reputation to candidate facts and then using them to select fused values [31]. Other approaches measure inconsistency using logical repair semantics or constraint-based metrics [4, 11, 27, 28]. Extensions such as PG-Schema bring similar validation capabilities to property graph models [2]. In summary, existing work largely focuses either on conceptual quality dimensions, isolated refinement tasks, or static KG evaluation. Most previous evaluation approaches and benchmarks focus on individual tasks rather than the combined effects of multiple integration stages. Existing KG quality evaluation approaches provide a useful starting point, but lack a focus on incrementally updating a KG through the integration of heterogeneous sources and evaluating the resulting end-to-end integration pipelines. In contrast, KGI-Bench supports suitable quality dimensions for the end-to-end evaluation of incremental KG integration pipelines. To the best of our knowledge, no previous work comparatively benchmarks the quality of different KG integration pipelines in this manner.
3
PRELIMINARIES
We first provide definitions of our notion of a knowledge graph and an integration pipeline. We then discuss requirements for evaluating KG integration pipelines and introduce the metrics of KGI-Bench to be defined in the next section.
3.1
Definitions
A knowledge graph (KG) is a structured representation of information where entities (such as people, places, or products) are
Evaluation of Pipelines for Data Integration into Knowledge Graphs
connected through relationships. An ontology is a formal specification of concepts and the relationships between them within a domain, providing the shared vocabulary and logical rules that underpin knowledge graphs and other semantic systems. In this work, we assume that KGs are represented in RDF. To limit the problem scope, we study data integration under a fixed KG ontology, leaving dynamic ontology changes for future work. We refer to individual nodes in the KG as entities, and their assigned entity types correspond to ontology classes. Following common RDF terminology, entities have a unique human-readable name or label, e.g., “Titanic (the movie)”. We use the term property for any RDF predicate. Properties that link one entity to another are called relations, while properties that link an entity to a literal value are called attributes. Unless the distinction matters, we use property to refer to both relations and attributes. Formally, the KG ontology 𝑂 = (𝐶, 𝑃) consists of a set 𝐶 of classes (entity types) and a set 𝑃 of properties. Each property 𝑝 ∈ 𝑃 is associated with ontology constraints, including a domain Dom(𝑝) ∈ 𝐶, a range Range(𝑝) ∈ 𝐶 for relations, or a datatype Datatype(𝑝) for attributes. A knowledge graph 𝐾𝐺 is defined with respect to 𝑂. We denote its entities by 𝐸 (𝐾𝐺) and its triples by 𝑇 (𝐾𝐺) ⊆ 𝐸 × 𝑃 × (𝐸 ∪ 𝐿), where 𝐿 is the set of literal values. For an entity 𝑒 ∈ 𝐸 (𝐾𝐺), we write 𝑡𝑦𝑝𝑒 (𝑒) ⊆ 𝐶 for its assigned types. We distinguish between relations, where the object is an entity, and attributes, where the object is a literal. We also use the term assertion for a triple stated in a KG. When such an assertion represents domain-level information, such as a film’s director or release date, we also refer to it as a fact. Thus, in the RDF setting used here, facts and assertions are realized as triples. Pipeline Definition Given a seed knowledge graph 𝐾𝐺 0 and one input source 𝑆, the integration task is to construct an updated graph 𝐾𝐺 1 = 𝜙 (𝐾𝐺 0, 𝑆). The source may overlap with the seed KG and may use different identifiers, labels, schemas, or representations, requiring tasks such as mapping, alignment, entity resolution, and fusion. We model a pipeline 𝜙 as a directed acyclic graph (DAG) 𝜙 = (𝑉𝜙 , 𝐸𝜙 ), where 𝑉𝜙 = {𝑇 𝐼 1, . . . ,𝑇 𝐼𝑘 } is a set of task implementations and 𝐸𝜙 denotes directed data-flow dependencies between tasks. Each task implementation 𝑇 𝐼𝑖 is a function 𝐹𝑖 : 𝐷𝑖𝐼 → 𝐷𝑖𝑂 with well-defined input and output data. A pipeline is valid if, for every task 𝑇 𝐼𝑖 , its required input 𝐷𝑖𝐼 is provided either by the initial inputs (𝐾𝐺 0, 𝑆, 𝑂) or by outputs of predecessor tasks in 𝜙. This ensures that the pipeline forms a well-defined composition of transformations that produces 𝐾𝐺 1 . Knowledge graph integration pipelines typically consist of combinations of the following core tasks: • Knowledge Extraction (KE): transform raw input into structured representations (e.g., triples or candidate entities). • Data Mapping (DM): map extracted data to the target ontology 𝑂.
• Schema Alignment / Ontology Mapping (SA/OM): align classes and properties across sources and the target ontology. • Entity Resolution (ER): identify correspondences between new entities and existing entities in the KG. • Entity Fusion (EF): merge aligned entities and resolve conflicting information. • Data Cleaning (DC): detect and correct inconsistencies or errors. • Knowledge Completion (KC): infer and add missing types, attributes, or relations. These tasks define a common integration workflow, but pipelines may differ in task ordering, implementation choices, and the subset of tasks applied, depending on the source format and integration strategy. The above definition covers a single source integration step. More complex integration scenarios arise by applying pipelines sequentially to multiple sources. For a sequence of sources 𝑆 1, . . . , 𝑆𝑛 , this yields 𝐾𝐺𝑖 = 𝜙𝑖 (𝐾𝐺𝑖 −1, 𝑆𝑖 ), 𝑖 = 1, . . . , 𝑛. Based on this sequential setting, we distinguish two integration settings. In the single-source type setting (SSP), all sources share the same data format (e.g., RDF, JSON, or text), and the same pipeline structure is applied repeatedly across sources. In the multi-source type setting (MSP), sources of different formats are integrated sequentially, requiring different pipeline structures or combinations of processing strategies across steps. Sequential integration in both settings introduces additional challenges such as error propagation, accumulation of inconsistencies, and dependencies of integration order. We later use this setting for incremental evaluation, where quality is analyzed after each integration step.
3.2
Benchmark Requirements and Metrics Overview
To evaluate knowledge graph (KG) integration pipelines it is not sufficient to assess the isolated performance of individual pipeline tasks but it is necessary to determine the quality of the knowledge graph updated by the complete pipeline. This is because errors introduced at early pipeline stages, such as extraction, mapping, or entity resolution, may propagate, interact, or accumulate during later stages, making task-level metrics insufficient to characterize overall pipeline behavior. A suitable benchmark for evaluating KG integration pipelines must therefore reflect the characteristics of end-to-end integration and support the computation of suitable metrics to determine the quality of the updated KG w.r.t. the data to be integrated. Furthermore, the benchmark should consider that KGs are updated incrementally so that an existing version of a KG is continuously updated to derive newer versions. A benchmark should thus provide an initial seed KG as well as several data sources that are to be integrated by pipelines to create newer versions of the KG. The benchmark should not be overly complex for the sake of interpretability but still be realistic requiring integration of heterogeneous data sources that can be dirty and contain overlapping data so that several of tasks mentioned abolve have to be executed in integration pipelines (knowledge extraction, entity resolution, etc.).
Marvin Hofer and Erhard Rahm
In KGI-Bench, we therefore evaluate pipelines at the level of Evaluation-restricted entity/triple sets. To avoid crediting the seed the integrated KG produced after each new source integration and, KG for information already present prior to integration, we restrict for the integration of multiple sources, after the execution of all evaluation to non-seed content. pipelines. To evaluate KG quality, we evaluate three complementary We define evaluation-restricted entity and triple sets as: quality dimensions: 𝐸 (𝐾𝐺𝑛 )eval = 𝐸 (𝐾𝐺𝑛 ) \ 𝐸 (𝐾𝐺𝑆 ), • KG coverage, capturing how completely source information 𝑇 (𝐾𝐺𝑛 )eval = 𝑇 (𝐾𝐺𝑛 ) \ 𝑇 (𝐾𝐺𝑆 ), is represented in the integrated KG; • KG correctness, capturing the semantic correctness of added and analogously 𝐸 (𝐾𝐺 𝑅(𝑛) )eval and 𝑇 (𝐾𝐺 𝑅(𝑛) )eval for the reference or changed information in the updated KG; and KG by excluding entities and triples already covered by the seed • KG consistency, capturing adherence of the updated KG to KG under the alignment. the formal constraints defined by the target ontology. Reference-based evaluation. A direct symbolic comparison beThese dimensions are orthogonal but interacting. A KG may extween 𝐾𝐺𝑛 and 𝐾𝐺 𝑅 is generally not possible, because equivalent hibit high coverage but low correctness, for example, by integrating real-world entities may be represented by different identifiers, lamany incorrect facts, or be logically consistent while still missing bels, or literal encodings. This is expected in our benchmark, for large parts of the expected information. Consequently, no single example, due to renamed namespaces in RDF sources, generated metric is sufficient so that meaningful evaluation requires their identifiers for JSON or text sources, and normalization differences in joint interpretation. literals. Therefore, evaluation must first establish correspondences In addition to these core quality dimensions, we consider auxbetween entities in the produced KG and entities in the reference iliary metrics such as structural statistics about the updated KG, KG. pipeline(s) execution runtime, and task-specific metrics such as We model this by an alignment relation recall and precision. These metrics support diagnosis, debugging, and performance analysis of pipelines but are not treated as pri𝐴𝑛 ⊆ 𝐸 (𝐾𝐺𝑛 ) × 𝐸 (𝐾𝐺 𝑅 ), mary indicators of end-to-end KG quality. KGI-Bench also supports where (𝑒, 𝑟 ) ∈ 𝐴𝑛 means that entity 𝑒 in the integrated graph the aggregation of different metrics in a single combined metric to corresponds to entity 𝑟 in the reference after integration step 𝑛. facilitate the comparison of different pipelines. The alignment relation 𝐴𝑛 may be obtained using different matchWe also provide benchmark datasets for the movie domain (KGI-Benching strategies depending on the available signals and the characMovie) including a seed KG, source datasets of three formats as teristics of the integrated sources. Typical strategies include exact well as a reference KG for evaluating coverage and correctness of identifier matching, label or literal similarity, provenance tracing the generated KG. across the benchmark construction process, and embedding-based entity similarity. In the context of the benchmark, alignment meth4 KGI-BENCH EVALUATION METRICS ods may therefore exploit both structural signals from the inteIn KGI-Bench, we assume that the benchmark datasets include a grated KG and provenance information from the dataset generation reference KG as a ground truth for the integrated KG that includes process. The evaluation framework itself is agnostic to the specific all input data. We will use the reference graph to determine coverage alignment method, requiring only the resulting correspondence and correctness (but not consistency) metrics. Given the difficulty to relation 𝐴𝑛 . In our experiments, we derive the alignment relation determine reference graphs and to underline the general usefulness using embedding-based similarity with sentence-transformer over of the proposed metrics beyond KGI-Bench, we will also discuss entity labels and a combination of matching strategies for literal alternate ways to determine coverage (using a so-called sourcevalues. based approach) and correctness (using a labeling-based method). Based on 𝐴𝑛 , we define the subset of reference entities repreWe first introduce notations for the definition of the metrics sented in the integrated graph as and the alignment between an integrated KG and the reference 𝐸𝑅𝐴 (𝑛) = {𝑟 ∈ 𝐸 (𝐾𝐺𝑅 ) | ∃𝑒 ∈ 𝐸 (𝐾𝐺𝑛 ) : (𝑒, 𝑟 ) ∈ 𝐴𝑛 }, graph. We then explain the metrics for the three quality dimensions coverage, correctness and consistency as well as auxiliary metrics. and analogously the subset of integrated entities aligned to the Finally, we propose a method to aggregate the metrics into a comreference as bined metric to allow an easier comparison and ranking of several 𝐸𝑛𝐴 = {𝑒 ∈ 𝐸 (𝐾𝐺𝑛 ) | ∃𝑟 ∈ 𝐸 (𝐾𝐺𝑅 ) : (𝑒, 𝑟 ) ∈ 𝐴𝑛 }. integration pipelines.
4.1
Evaluation Notation and Reference Alignment
Let 𝐾𝐺 0 denote the seed knowledge graph, let 𝑆 1, . . . , 𝑆𝑛 be the source datasets integrated sequentially, and let 𝐾𝐺𝑛 be the resulting graph after integrating the first 𝑛 sources. Furthermore, let 𝐾𝐺 𝑅 denote the benchmark reference KG. For a dataset or graph 𝑑, we write 𝐸 (𝑑) for its entities and 𝑇 (𝑑) for its RDF triples. We use 𝐾𝐺 𝑅(𝑛) ⊆ 𝐾𝐺 𝑅 to denote the subset of the reference KG that is relevant after the first 𝑛 integration steps.
Triple Matching Function. To compare triples, we additionally require literal equivalence. Let ≈𝐿 denote an equivalence relation over literals, e.g., exact equality, normalized string equality, normalized dates, or numerically equivalent values. For triples 𝑡𝑛 = (𝑠𝑛 , 𝑝𝑛 , 𝑜𝑛 ) ∈ 𝑇 (𝐾𝐺𝑛 ),
𝑡𝑅 = (𝑠𝑅 , 𝑝 𝑅 , 𝑜 𝑅 ) ∈ 𝑇 (𝐾𝐺𝑅 ),
we say that 𝑡𝑛 matches 𝑡𝑅 , written match(𝑡𝑛 , 𝑡𝑅 ), iff all of the following hold: (1) (𝑠𝑛 , 𝑠𝑅 ) ∈ 𝐴𝑛 , (2) 𝑝𝑛 = 𝑝 𝑅 ,
Evaluation of Pipelines for Data Integration into Knowledge Graphs
(3) either (𝑜𝑛 , 𝑜 𝑅 ) ∈ 𝐴𝑛 for entity-valued objects, or 𝑜𝑛 = 𝑜 𝑅 or 𝑜𝑛 ≈𝐿 𝑜 𝑅 for literal-valued objects. Using this definition, the set of matched reference triples is 𝑇𝑅𝐴 (𝑛) = {𝑡𝑅 ∈ 𝑇 (𝐾𝐺𝑅 ) | ∃𝑡𝑛 ∈ 𝑇 (𝐾𝐺𝑛 ) : match(𝑡𝑛 , 𝑡𝑅 )}, and the set of matched produced triples is 𝑇𝑛𝐴 = {𝑡𝑛 ∈ 𝑇 (𝐾𝐺𝑛 ) | ∃𝑡𝑅 ∈ 𝑇 (𝐾𝐺 𝑅 ) : match(𝑡𝑛 , 𝑡𝑅 )}. This yields a two-stage evaluation procedure: first align entities between 𝐾𝐺𝑛 and 𝐾𝐺 𝑅 , and then compute coverage and correctness metrics using the resulting correspondences.
4.2
KG Coverage
4.3
Coverage measures how completely the source information to be integrated is represented in the integrated KG. We distinguish between entity coverage and triple or fact coverage. Entity coverage. Entity coverage measures how many expected reference entities are represented in the integrated graph with the expected semantic type. Accordingly, a reference entity is counted as covered only if there exists an aligned entity in the integrated graph that is assigned the reference type, since correct typing is important for many downstream tasks. We thus define the set of reference entities covered in the integrated KG as follows: 𝐸𝑅𝐴,𝑇 (𝑛) = { 𝑟 ∈ 𝐸 (𝐾𝐺 𝑅 ) | ∃𝑒 ∈ 𝐸 (𝐾𝐺𝑛 ) : (𝑒, 𝑟 ) ∈ 𝐴𝑛 ∧ 𝑡𝑦𝑝𝑒 (𝑟 ) ⊆ 𝑡𝑦𝑝𝑒 (𝑒) }. Note that we do not require that the semantic types of aligned entities are equal but only that the type of the reference entity is included in the set of types of an entity in the integrated KG. This is because integrated entities may acquire several valid types during integration, for example through type inference or completion. For instance, an entity extracted as an Actor may additionally obtain the more general type Person through ontology hierarchy inference, while still correctly representing the reference entity. We therefore require that all reference types are preserved while allowing additional compatible types. Entity coverage is then defined as Cov𝐸 (𝐾𝐺𝑛 ) =
|𝐸𝑅𝐴,𝑇 (𝑛) ∩ 𝐸 (𝐾𝐺 𝑅(𝑛) )𝑒𝑣𝑎𝑙 | .
|𝐸 (𝐾𝐺 𝑅(𝑛) )𝑒𝑣𝑎𝑙 |
considering only the reference entities not already included in the seed KG. Fact coverage. Fact coverage measures how many expected reference triples are represented in the integrated graph: Cov𝑇 (𝐾𝐺𝑛 ) =
𝑇 (𝐾𝐺𝑛 ) is considered correct if it can be derived from or matched to a corresponding fact in 𝑇 (𝑆𝑖 ), and for coverage, if for a 𝑇 (𝑆𝑖 ) a matching 𝑇 (𝐾𝐺𝑛 ) is aligned. This evaluation assumes that the input data is correct and complete with respect to the evaluated facts. While this assumption is reasonable in controlled benchmarks or curated datasets, it may not hold for noisy or real-world sources, where errors can propagate into the KG. Furthermore, this evaluation is inherently local to each source and does not capture cross-source consistency or completeness, making it complementary to reference-based coverage metrics and global consistency checks.
|𝑇𝑅𝐴 (𝑛) ∩ 𝑇 (𝐾𝐺 𝑅(𝑛) )𝑒𝑣𝑎𝑙 | |𝑇 (𝐾𝐺 𝑅(𝑛) )𝑒𝑣𝑎𝑙 |
.
Source-based evaluation. Coverage can also be assessed without reference KG by directly evaluating the integrated KG against the source data to be integrated. In this setting, the source 𝑆𝑖 itself serves as the ground truth, and the evaluation verifies whether the information extracted and integrated into the KG is correctly entailed from the source. Formally, let 𝑇 (𝑆𝑖) denote the set of source facts (after extraction and normalization). 𝐴 produced triple 𝑡𝑛 ∈
KG Correctness
The integration of source entities and triples into a KG can lead to many errors, reducing the quality of the resulting KG. Possible errors include unresolved duplicates, false positive entity merges, wrong relation mappings, incorrectly linked literals, or spurious facts introduced during extraction and fusion. We use the metrics entity correctness and triple correctness to determine the share of added entities and triples in the integrated KG that are semantically correct relative to the reference KG. The metrics are precisionoriented, as they quantify the fraction of produced entities/triples that are correct. Entity Correctness. Entity precision measures for the newly integrated entities the share of correct entities, i.e., entities that are correctly aligned to reference entities and assigned the appropriate entity types. The produced entities aligned with the reference KG are defined as 𝐸𝑛𝐴,𝑇 = { 𝑒 ∈ 𝐸 (𝐾𝐺𝑛 ) | ∃𝑟 ∈ 𝐸 (𝐾𝐺𝑅(𝑛) ) : (𝑒, 𝑟 ) ∈ 𝐴𝑛 , 𝑡𝑦𝑝𝑒 (𝑟 ) ⊆ 𝑡𝑦𝑝𝑒 (𝑒) } The entities in this set can still contain duplicates, i.e., entities aligned with the same reference entity. Duplicates are problematic because they indicate unresolved entity resolution errors. As a result, information about the same real-world object is split across multiple KG nodes, which not only leads to redundancy but can also cause wrong query results and assertions. To avoid counting all such duplicates as correct, we count the reference entities represented in the integrated KG. We define the aligned reference entities as 𝐴,𝑇 𝐸𝑅,𝑛 = {𝑟 ∈ 𝐸 (𝐾𝐺 𝑅(𝑛) )𝑒𝑣𝑎𝑙 | ∃𝑒 ∈ 𝐸𝑛𝐴,𝑇 : (𝑒, 𝑟 ) ∈ 𝐴𝑛 }.
Then the entity correctness becomes: 𝐶𝑟𝑐𝑡𝐸 (𝐾𝐺𝑛 ) =
𝐴,𝑇 |𝐸𝑅,𝑛 |
, |𝐸 (𝐾𝐺𝑛 )𝑒𝑣𝑎𝑙 | The use of reference entities in the nominator ensures that each correctly aligned entity can contribute at most once, while duplicate produced entities increase the count in the denominator, thereby penalizing entity precision. We will also report the duplicate rate as an auxiliary diagnostic metric for explaining unresolved entity resolution errors. As an auxiliary metric, we also determine the degree of duplicates by a so-called duplicate rate. A duplicate is present for a reference
Marvin Hofer and Erhard Rahm
entity 𝑟 whenever the set Dup(𝑟 ) = {𝑒 ∈ 𝐸 (𝐾𝐺𝑛 ) | (𝑒, 𝑟 ) ∈ 𝐴𝑛 }. has more than one element. We measure the duplicate rate as Í 𝑟 ∈𝐸 (𝐾𝐺𝑅 ) max(0, |Dup(𝑟 )| − 1) DupRate(𝐾𝐺𝑛 ) = . |𝐸 (𝐾𝐺𝑛 )| Fact Correctness. We again use the alignment relation and triple matching function introduced above. Intuitively, a produced triple is accurate (correct) if it can be matched to a corresponding triple in the reference graph. As for entity precision, we have to account for duplicate-like triples that match with the same triple of the reference graph. This leads to the following definition of fact precision: 𝐶𝑟𝑐𝑡𝑇 (𝐾𝐺𝑛 ) =
• Disjointness violations: entities assigned to mutually disjoint classes Example: an entity typed both as Film and Person. |𝑒 ∈ 𝐸|∃𝑐1, 𝑐2 ∈ 𝑇 (𝑒) : Disjoint(𝑐1, 𝑐2)| 𝑂 𝐷𝑇 = |𝐸|
𝐴 | |𝑇𝑅,𝑛
|𝑇 (𝐾𝐺𝑛 )𝑒𝑣𝑎𝑙 |
.
Labeling-based evaluation. While the above formulation relies on an explicit reference KG and an alignment relation 𝐴𝑛 , correctness can alternatively be assessed through direct labeling of entities or triples. In this setting, individual entities or triples in the integrated KG are annotated as correct or incorrect using human experts, LLMbased judgments, or hybrid approaches. This eliminates the need for reference alignment and avoids errors introduced by imperfect matching. Formally, let 𝐿𝑛 ⊆ 𝑇 (𝐾𝐺𝑛 ) denote the subset of labeled triples, together with a labeling function 𝑙 : 𝐿𝑛 → {0, 1} indicating correctness. Correctness can then be estimated as the fraction of correctly labeled triples within 𝐿𝑛 . In practice, labeling is expensive and thus not scalable. It is therefore often performed on samples only, yielding statistical estimates of correctness rather than measurements. Another limitation is that duplicates are not easily considered so that precision values may become overly high. Alternatively, a source-based approach could verify whether a produced KG fact is semantically supported by the source content itself, for example using natural language inference (NLI) methods or LLM-based judges. This is particularly relevant for unstructured or semi-structured sources, where extracted facts may not correspond to exact symbolic matches but can still be supported by the original source text.
4.4
one complementary quality dimension for evaluating the outputs of heterogeneous KG integration pipelines. The consistency metrics defined below capture several common types of ontology violations observed in KG integration pipelines. This set is not intended to be exhaustive. Other ontology constraints, such as cardinality restrictions, functional properties, or key constraints, may also be evaluated depending on the ontology and application context. Let type(𝑒) denote the asserted types of entity 𝑒, and let Dom(𝑝), Range(𝑝), and Datatype(𝑝) denote the ontology-defined domain, range, and datatype of property 𝑝. We compute the following four violation ratios:
KG Consistency
Consistency measures whether the integrated KG conforms to the structural and semantic constraints defined by the target ontology. In contrast to coverage and correctness, which compare the produced graph against expected reference content, consistency evaluates whether the resulting graph is internally well-formed with respect to ontology-defined typing, relation usage, and literal constraints. Ontology consistency has also been analyzed in prior work (e.g. RDFUnit [25], SHACL or SHeX [39]), but it has not yet been investigated as one complementary quality dimension within a benchmark for evaluating end-to-end KG integration pipelines, alongside coverage and correctness. In contrast to ontology-evolution approaches such as [3], which primarily evaluate conceptual or structural changes in existing ontologies, we study consistency as
• Domain and range violations: relations connecting entities whose types do not match the ontology specification Example: using directedBy to link two Person entities instead of Film → Person. (𝑠, 𝑝, 𝑜) ∈ 𝑅 ∃𝑐𝑠 ∈ 𝑇 (𝑠) : Disjoint 𝑐𝑠 , Dom(𝑝) 𝑂𝐷 = |𝑇 (𝐾𝐺𝑛 )| (𝑠, 𝑝, 𝑜) ∈ 𝑅 ∃𝑐𝑜 ∈ 𝑇 (𝑜) : Disjoint 𝑐𝑜 , Range(𝑝) 𝑂𝑅 = |𝑇 (𝐾𝐺𝑛 )| • Relation direction violations: inverse use of relations relative to ontology definitions Example: asserting Person → Film for a relation defined as Film → Person. {(𝑠, 𝑝, 𝑜) ∈ 𝑇 (𝐾𝐺𝑛 ) | ∃𝑐𝑠 ∈ T(𝑠), ∃𝑐𝑜 ∈ T(𝑜) : 𝑂 𝑅𝐷 =
𝑐𝑠 ⊑ Range(𝑝) ∧ 𝑐𝑜 ⊑ Dom(𝑝)} |𝑇 (𝐾𝐺𝑛 )|
• Datatype and format violations: literal values violating declared datatypes or formats Example: a runtime stored as a free-text string instead of a numeric value, or dates in a different format. (𝑠, 𝑝, ℓ) ∈ 𝐿 ¬ValidDatatype ℓ, Datatype(𝑝) 𝑂 𝐿𝑇 = |𝐿| (𝑠, 𝑝, ℓ) ∈ 𝐿 ¬ValidFormat ℓ, Format(𝑝) 𝑂 𝐿𝐹 = |𝐿| If needed, a similar metric can be defined for format constraints such as year, duration, or currency normalization. For reporting, these violation ratios can either be presented directly or transformed into compliance scores 𝐶𝑖 = 1 − 𝑂𝑖 , so that higher values uniformly indicate better consistency.
4.5
Auxiliary Metrics
In addition to the end-to-end quality metrics discussed above, a variety of auxiliary metrics are commonly reported in KG construction and integration pipelines. These metrics provide useful insights into structural characteristics, resource consumption, and task-specific performance of individual pipeline components. However, they are not suitable as end-to-end quality measures, as they do not directly
Evaluation of Pipelines for Data Integration into Knowledge Graphs
reflect the semantic correctness, completeness, or logical usability of the resulting KG. In particular, increases in size, density, or computational efficiency do not necessarily correspond to improvements in KG quality and may even mask integration errors. We therefore exclude these metrics from our end-to-end quality assessment and report them only as diagnostic and explanatory indicators for pipeline analysis, debugging, and scalability evaluation.
KGI-Bench supports a two-step aggregation of metrics. We first normalize and aggregate selected metrics per group (e.g. quality dimension) and then determine a weighted average of the group metrics to obtain the overall aggregated score of a pipeline. Each group metric is the average of the normalized metrics 𝑀𝑛𝑖 (mapped to interval [0,1], higher is better) of the selected individual metrics 𝑀 𝑖 per group:
Statistical Metrics. These metrics capture the structural properties of the generated KG, giving a baseline view of its size and complexity (more classes or properties). We measure the structural characteristics of the generated KG:
𝑀𝑛 1 + · · · + 𝑀𝑛𝑘 . 𝑘 The total aggregated evaluation score 𝑀total (𝑝) of a pipeline 𝑝 is the weighted average of the group metrics:
• SFC Fact Count: Number of distinct triples (facts). • SEC Entity Count: Number of distinct entities (nodes). • SRC Relation Count: Number of distinct relation names. • STC Type Count: Number of distinct entity classes (types). • SUT Count of untyped entities that remain without a class, e.g., because none of their extracted or mapped properties match the ontology. In such cases, type inference may not be able to assign a class leading to isolated label-only entities. • SD Graph Density: Ratio of existing relations to the maximum possible number of relations between entities, indicating overall connectivity. Identifying most relations results in a KG with high density while a KG with mostly isolated entities has low density. Resource Metrics. These metrics capture the computational cost and efficiency of generating the KG (Quality-of-service). • QD Duration: Total execution time of the pipeline in seconds. • QM Max Memory: Peak memory consumption during runtime. • QC Additional Costs: Non-computational overhead such as API usage, cloud hosting, or power consumption. Task Metrics. For each task in a pipeline there can be a set of specific evaluation metrics. In addition to the duplicate rate already introduced in subsection 4.3, KGI-Bench considers: • REM Entity Matching: Precision and recall of expected vs. actual entity correspondences in entity resolution tasks [8]. • ROM Ontology Matching: Precision and recall of expected vs. actual relation/property correspondences in ontology matching tasks. • REL Entity Linking: Coverage of links between entities in the KG and external reference identifiers (e.g., ensuring each root entity is linked to at least one external reference). • RRL Relation Linking: Correctness of links between input features (e.g., JSON keys) and ontology relations, where gold mappings are available.
4.6
Aggregation and Ranking Method
To better compare and rank different pipelines, it is desirable to aggregate the different metrics in fewer scores or even a single score per pipeline. A simple way is to determine the F1-scores for entities and triples as the harmonic means of their coverage (recall) and precision metrics. To combine more metrics in a single score,
𝐺𝑀𝑖 =
𝑀total (𝑝) =
𝑗 ∑︁
𝑤𝑖 · 𝐺𝑀𝑖 (𝑝),
𝑖=1
𝑗 ∑︁
𝑤𝑖 = 1,
𝑖=1
When focusing on evaluating the three quality dimensions KG coverage, correctness, and consistency, we obtain: 𝑀total = 𝛼 · 𝐺𝑀cov + 𝛽 · 𝐺𝑀acc + 𝛾 · 𝐺𝑀cons where 𝛼, 𝛽, 𝛾, 𝛿 are weights that can be adjusted depending on the user’s goals (e.g., coverage vs consistency). We will use this approach in the evaluation in Section 7.2.
5
KGI-BENCH-MOVIE
The KGI-Bench evaluation metrics can be applied to integration tasks for knowledge graphs of different domains and application purposes. For specific and comparable evaluation results, we need however a well-defined KG and source datasets to be integrated of a certain domain. In this section we therefore outline an instantiation of KGI-Bench for the movie domain (KGI-Bench-Movie) and how we created the datasets. Benchmark datasets for other domains can be constructed in an analogous way. The next section outlines example integration pipelines that we use in our evaluation with KGI-Bench-Movie (Section 7). The KGI-Bench-Movie datasets cover films and their related persons and companies. The domain provides a manageable yet realistic setting with heterogeneous data formats, partial overlap between sources, and schema-level heterogeneity. While existing benchmarks such as Text2KGBench [33] evaluate isolated text-toKG extraction tasks, KGI-Bench-Movie targets incremental endto-end KG integration across heterogeneous RDF, JSON, and text sources, including challenges such as entity resolution, fusion, and ontology consistency. While the benchmark is domain-specific, it represents a general class of incremental KG integration scenarios across heterogeneous sources of three formats (RDF, JSON, and text). We first describe the domain and KG ontology before providing details about the datasets (reference KG, seed KG, sources with overlapping entities) and their generation. Domain and Ontology. Our benchmark for the movie domain includes entities such as films, actors, directors, and production companies. We manually curated a target ontology for this domain comprising: • Three core classes: Film, Person, and Company, serving as the backbone of the domain model. • 25 properties (object relations or datatype properties, and counting the RDF label and type property), defined using
Marvin Hofer and Erhard Rahm
Figure 1: Ontology/Schema graph of classes film, person, company with their properties (relations and attributes).
OWL, RDF Schema, and SKOS, enabling consistent links across heterogeneous sources. • Annotations such as rdfs:label and skos:altLabel (to support lexical variation) and owl:equivalentClass or owl:equivalentProperty) that facilitate schema alignment. • Integrity axioms, including owl:disjointWith (to catch invalid class overlaps, e.g., a Film typed as Person) and owl:maxCardinality (to follow rules, e.g., one runtime per film). A visualization of this graph schema (ontology) is shown in Figure 1. A well-defined ontology is essential for the integration process because it provides the common target schema against which heterogeneous sources can be aligned. In the linking step, ontology classes and relations guide entity and schema matching, ensuring that different source fields map to the correct concepts. For cleaning, constraints and disjointness axioms allow the detection of errors such as a film mistakenly typed as a person or multiple runtimes assigned to the same movie. Finally, during completion, the ontology defines where additional values can be added consistently, for example, enriching a person entity with missing birthPlace or linking a film to its director. In short, the ontology acts as the semantic backbone that enables heterogeneous data to be reconciled into a coherent knowledge graph. Although the ontology provides a clear target schema, integration remains challenging due to ambiguity and inconsistency in the sources. For example, some JSON infoboxes use the field name "date" without specifying whether it denotes a person’s birth or death date, which are distinguished in the ontology. Ambiguities also arise among role-related person or company properties for a film, e.g, producer as person vs production as company. Properties may also use different data types. Runtime may appear in minutes, hours, or as free-text strings (“2h 15m”), and financial attributes such as budget, gross, and revenue are reported with inconsistent currencies and not always distinguished across sources. Datasets Our benchmark is built from a set of interrelated data artifacts that provide the basis for constructing and evaluating
pipelines. Starting from a reference graph, we derive seed and source datasets in multiple formats (RDF, JSON, and text), each designed to expose specific integration challenges. These artifacts vary in coverage, structure, and representation, thereby reflecting the heterogeneity typical of real-world scenarios. Reference KG. To establish a basis for evaluation, we first derive a reference by recursively collecting data for the film entities from DBpedia, and their related persons and companies, whenever such links are provided by the eight ontology relations. We restrict our benchmark to 10,000 films rather than the full set of 150,000 available in DBpedia (the final size is a bit lower due to cleaning and generating overlaps). This size already yields a knowledge graph of substantial complexity once related persons and companies are included, while keeping experiments computationally feasible and reproducible. It offers sufficient semantic variety to capture typical integration challenges without introducing unnecessary volume. For development and reduced resource requirements, we provide two smaller benchmark versions with only a total of 100 and 1,000 film entities. From the reference graph, we generate multiple entity splits to emulate realistic multi-source settings. Specifically, we partition the set of all films into four subsets (seed and three sources) of 25% of the entities each. To introduce controlled redundancy, every pair of subsets (seed, sources) shares around 5% of their film entities (30% including persons and companies). This controlled overlap reflects the partial coverage commonly observed in real-world sources, where new sources usually cover existing entities in the KG. Seed/Source type RDF. The seed KG is defined as the first split of the reference graph. To generate the corresponding source RDFs, we create shaded versions of the remaining splits by renaming entity identifier namespaces. This ensures that the same real-world entities are represented with different IRIs across splits, thereby requiring entity resolution during integration. # S o u r c e RDF−KG rdf:Titanic a rdf:Film ; rdfs:label "Titanic" ; rdf:actor rdf:DiCaprio . rdf:Diamonds a rdf:Film ; rdfs:label "Diamonds" ; rdf:actor rdf:Douglas .
Source type JSON. For each film entity, we generate nested JSON records derived from one subgraph per film, including referenced person and company information. Each record contains key–value pairs describing the film, such as title, actors, genre, or production company. JSON is a suitable source format for testing pipelines, since it is widely used on the Web, comes with less structural richness than RDF, and requires additional mapping to be aligned with the ontology. # JSON Document { "title" : "Inception", "runtime" : "8880.00", "starring" : [ { "name" : "Leonardo DiCaprio", "birthYear" : ... } ... ] ... }
Source type Text. As a complementary unstructured source, we include DBpedia abstracts for all film entities. These textual descriptions provide narrative information mentioning films, people,
Evaluation of Pipelines for Data Integration into Knowledge Graphs
and companies without explicit schema structure. Text is an important source type because it reflects the reality that much of the Web’s knowledge is available only in natural language and requires information extraction to be transformed into a knowledge graph. # T e x t Document Titanic is a 1997 American epic historical romance film written and directed by James Cameron . Incorporating both historical and fictional aspects, it is based on accounts of the sinking of RMS Titanic in 1912. Leonardo DiCaprio and Kate Winslet star as members of...
Complementarity of Sources. Together, RDF, JSON, and text capture the three principal degrees of structure encountered in realworld data: highly structured (RDF), semi-structured (JSON), and unstructured (text). This diversity ensures that the benchmark exercises pipelines across the full spectrum of integration challenges, from ontology alignment and schema mapping to entity resolution and information extraction. Supplementary Data. In addition to the reference, seed, and source datasets, we provide supplementary data that supports evaluation and error analysis. First, we include metadata about the expected matches, i.e., the overlapping entities introduced during the dataset splitting. These records specify the entity type (e.g., Film) together with explicit links of the form id1 = id2, which serve as ground truth for entity resolution tasks. Second, we provide curated lists of verified source entities. These lists indicate which entities should appear in the integrated KG, enabling evaluators to check for missing or erroneously introduced entities. Together, these supplementary resources facilitate a more fine-grained assessment of pipeline performance beyond the structural and semantic properties of the resulting graphs. We end the section with two additional examples: (i) a targeted Seed KG and (ii) the resulting integrated RDF KG obtained after incorporating the three previously shown source snippets into the seed. Color highlights indicate the origin of each entity (i.e., from the seed or from a specific source). In this example, no conflicting attribute values occur during integration. # S e e d RDF−KG kg:Titanic a kg:Film ; rdfs:label "Titanic" ; kg:runtime "11700.00"^^xsd:double . kg:actor kg:DiCaprio . kg:Matrix a kg:Film ; rdfs:label "The Matrix" ; kg:release "1999"^^ x s d : gDate ; kg:genre "science fiction", "action film" ; kg:writer kg:Wachowskis; kg:director:Wachowskis ; kg:actor kg:Reeves . # I n t e g r a t e d R e s u l t RDF−KG kg:Titanic a kg:Film ; rdfs:label "Titanic" ; kg:runtime "11700.00"^^xsd:double . kg:release "1997"^^xsd:gDate ; kg:genre "epic", "historical romance" ; kg:writer txt:Cameron ; kg:director txt:Cameron . kt:actor kg:DiCaprio , txt:Winslet . kg:Matrix a kg:Film ; rdfs:label "The Matrix" ; . . . kg:actor kg:Reeves . ex:Inception a kg:Film ; rdfs:label "Inception" ;
Figure 2: RDF single-source pipeline layouts used in the evaluation. TC=Type Completion.
Figure 3: JSON single-source pipeline layouts used in the evaluation. TC=Type Completion
kg:runtime "8800"^^xsd:double . kg:actor kg:DiCaprio . ex:Diamonds a kg:Film ; rdfs:label "Diamonds" ; kg:actor ex:Douglas .
6
EVALUATED PIPELINES
To apply the proposed benchmark, we define a set of knowledge graph integration pipelines operating on the source formats supported by KGI-Bench-Movie: RDF, JSON, and text. The pipelines are not intended to be optimal solutions, but serve as representative integration strategies with different design choices based on existing task implementations or LLM functionality. For each source format, we define two pipeline variants: (i) a baseline pipeline using standard techniques, and (ii) a variant in which selected tasks are replaced by LLM-based implementations. All pipelines have been specified and executed with the open-source tool KGpipe [18]. Figures 2 to 4 illustrate the corresponding six pipeline layouts for RDF, JSON, and text sources, respectively. All pipelines follow the general workflow introduced in Section 3.1, comprising
Marvin Hofer and Erhard Rahm
6.2
Semi-Structured Sources (JSON)
Semi-structured data introduces complexity through implicit or evolving schema, nested records, and heterogeneous field names. Converting JSON, XML, or CSV into graph-compatible form requires therefore careful parsing and normalization, often combined with type inference to recover implicit semantics. A central challenge here is schema interpretation and mapping [5, 45], as the source structure may only partially reflects the target ontology. A typical pipeline is: (i) Parsing and Normalization → (ii) Data Mapping (to RDF) → (iii) Type Inference / Schema Alignment → (iv) Entity Resolution → (v) Fusion → (vi) Cleaning. For the JSON format, we also have two layouts with a configuration each (see Figure 3). • 𝐽𝑆𝑂𝑁𝑏𝑎𝑠𝑒 maps JSON data into a generic RDF graph, each key as a generic predicate URI, and types are constructed from the current path keys. The implementation attempts to identify rdfs:label values from key–value pairs in JSON objects and distinguishes between shallow objects (treated as entities with labels) and literals (e.g., dates, numbers) using regular expressions. The constructed RDF is then integrated into the seed KG following the same steps as in 𝑅𝐷𝐹𝑏𝑎𝑠𝑒 . • 𝐽𝑆𝑂𝑁𝑙𝑙𝑚 avoids explicit mapping generation and instead prompts an LLM to directly output ontology-compliant RDF triples from the input JSON document. The LLM receives both the JSON data and the KG ontology as input and generates triples aligned with the ontology. Unlike approaches that require generating and selecting among candidate mappings, this strategy applies uniformly across all documents, thereby averaging out quality fluctuations in the LLM outputs over the dataset. This removes the need for manual or heuristic mapping selection, though it still introduces variability at the individual document level.
Figure 4: Text single-source pipeline layouts used in the evaluation. TC=Type Completion tasks such as knowledge extraction, data mapping, schema/ontology alignment, entity resolution, and fusion to address common integration challenges such as schema heterogeneity and ambiguity in extraction and matching. Depending on the source format and implementation, some of these steps may be simplified, combined, or omitted. Pipelines combine established tools with lightweight custom transformations that ensure interoperability between tasks by converting outputs into the required input formats for subsequent tasks. The LLM-based variants replace selected steps such as extraction, mapping, or ontology matching with generative approaches. In the following we discuss the pipelines for the three input formats.
6.1
Structured Sources (RDF)
For structured sources, integration primarily focuses on schema alignment and entity resolution, as data is already available in a graph-like representation. A typical pipeline is: (i) Data Mapping (to RDF for relational input) → (ii) Schema Alignment → (iii) Entity Resolution → (iv) Entity Fusion → (v) Cleaning → (vi) KG completion (optional). We evaluate two RDF pipeline layouts, each with one configuration (see Figure 2): • 𝑅𝐷𝐹𝑏𝑎𝑠𝑒 applies the established graph alignment method PARIS [42] (executed in a Docker container) to produce matches between entities in the seed and source RDF graphs. A fusion algorithm with a first-value preference is then applied for matching entities: entity identifiers are resolved from source to seed, and for fusable relations, the first available value (typically from the source) is selected. Afterwards, type information on entities is inferred based on their current properties corresponding domain/range specs. • 𝑅𝐷𝐹𝑙𝑙𝑚 follows the same structure as 𝑅𝐷𝐹𝑏𝑎𝑠𝑒 , but determines relation alignment (ontology matching) with the help of a Large Language Model (LLM). The LLM is used to match and map relations between the source and seed RDF graphs, based on sampled triples from each KG. PARIS is then applied for entity matching, followed by the same first-value fusion strategy.
6.3
Unstructured Sources (Text)
Unstructured sources such as text and web pages suffer from ambiguity, noise, and limited inherent structure. The main burden of integration falls on natural language processing, which transforms raw text into structured candidate triples. Named entity recognition, relation extraction, and co-reference resolution form the entry point, followed by linking entities and predicates to the target ontology. Knowledge extraction from text is known to be error-prone and highly sensitive to domain, extraction model, and linking strategy, with errors propagating downstream [7, 10, 30, 40]. Uncertainty introduced at the extraction stages propagates through the entire pipeline and can limit integration quality. A typical pipeline is: (i) Knowledge Extraction → (ii) Entity/Relation Linking → (iii) Schema Alignment → (iv) Entity Resolution → (v) Cleaning → (vi) Completion. We use almost similar pipeline layouts for text sources, but with two different configurations (see Figure 4): • 𝑇 𝐸𝑋𝑇𝑏𝑎𝑠𝑒 extracts triple patterns (entity surface forms) from the input text using the tool OpenIE. Entities are
Evaluation of Pipelines for Data Integration into Knowledge Graphs
7
linked with DBpedia Spotlight1 , which maps mentions to the seed KG. Mentions not present in the seed are assigned new identifiers in a separate namespace. Relations are then linked using a custom embedding-based relation linker that maps text spans to ontology relations. The constructed KG is then integrated into the seed KG using the alignment and fusion tasks of 𝑅𝐷𝐹𝑏𝑎𝑠𝑒 . • 𝑇 𝐸𝑋𝑇𝑙𝑙𝑚 replaces OpenIE with an LLM-based triple extraction component, which generates surface-form triples directly from the input text. Entity linking, relation linking, and fusion are performed as in 𝑆𝑆𝑃𝑇 𝐸𝑋𝑇𝑏 .
Table 1: Pipeline result statistics: FC=Fact Count, EC=Entity Count, RC=Relation Count, TC=Type Count, UT=Un-Typed Entities, D=Duration in sec, M=Memory in Gigabyte. * the 𝐽𝑆𝑂𝑁𝑙𝑙𝑚 and 𝑇 𝐸𝑋𝑇𝑙𝑙𝑚 pipelines are only executed for the 1k version of the benchmark.
EVALUATION
Our evaluation mainly aims at demonstrating the usability and usefulness of the KGI-Bench benchmark, by comparatively evaluating the metrics introduced in Section 4 for the datasets and incrementally executed pipelines of the movie domain. In addition of determining the relative effectiveness of different pipeline we also aim at identifying integration problems visible through the produced KG We focus on integrating three sources (data splits) into the seed KG of KGI-Bench-Movie by incrementally executing three pipelines to generate a final KG that can be compared to the reference KG. For the three input data formats, we apply the six single-source type pipelines (SSPs) of Section 6 when all splits have the same format and six multi-source type pipelines (MSP) when each split is of a different format. The MSP pipelines use the base version of the SSP pipelines. For example, the MSP pipelines abbreviated RJT indicates the MSP of the SSP sequence RDFbase→JSONbase→TEXTbase. All pipelines were executed on a machine equipped with an AMD Ryzen 9 9850HX processor (16 cores, 32 threads), 64 GB of main memory, a 1 TB SSD, and an NVIDIA RTX 4070 GPU. The availability of the GPU significantly increased the throughput of embeddingbased task implementations. To balance quality and runtime, the LLM-based pipelines use OpenAI’s gpt-5-mini-2025-10, with API costs of RDFllm=0.1€, JSONllm=9.9€, and TEXTllm=2.3€. Prompts are linked in the project repository. The LLM-based pipelines for JSON and text had to be executed on the smaller 1k version of the benchmark to limit execution time and monetary expenses. All LLM-based tasks use OpenAI’s gpt-5-mini model (API version as of October 2025), with fixed prompts and zero-shot inference. We apply fixed thresholds2 across all pipeline configurations to ensure consistency of evaluation and limit the number of pipeline executions: PARIS matches entities at a similarity threshold of 0.95 and relations at 0.8, LLM OM matches at 1.0, DBpedia Spotlight links at 0.8, and an embedding-based relation linker (EmbRL) at 0.8. We next evaluate the integration pipelines using the metrics introduced in Section 4 and also analyze their incremental behavior. Finally, we present comparative rankings of the 12 pipeline combinations.
1 DBpedia Spotlight is suitable for entity recognition and linking, because DBpedia widely covers the domain of interest. 2 Thresholds were selected based on preliminary experiments exploring a range of parameter settings
Pipeline
FC
EC
RC
TC
UT
D(s)
M(GB)
Seed 1k Seed 10k Ref. 1k∗ Ref. 10k
16,417 123,686 63,359 336,002
2,793 19,527 8,935 47,706
25 25 25 25
3 3 3 3
-
-
-
𝑅𝐷𝐹𝑏𝑎𝑠𝑒 𝑅𝐷𝐹𝑙𝑙𝑚 𝐽𝑆𝑂𝑁𝑏𝑎𝑠𝑒 𝐽𝑆𝑂𝑁𝑙𝑙𝑚 * 𝑇 𝐸𝑋𝑇𝑏𝑎𝑠𝑒 𝑇 𝐸𝑋𝑇𝑙𝑙𝑚 *
256,528 327,714 242,319 49,787 133,883 24,134
47,258 47,013 46,657 7,668 23,946 3,608
25 25 25 25 25 25
3 3 3 3 3 3
3,500 273 4,801 48 72 3
65 222 77 7,025 1,004 1,963
6.3 6.1 4.3 4.8 19.8 10.4
RJT𝑏𝑎𝑠𝑒 RTJ𝑏𝑎𝑠𝑒 JRT𝑏𝑎𝑠𝑒 JTR𝑏𝑎𝑠𝑒 TJR𝑏𝑎𝑠𝑒 TRJ𝑏𝑎𝑠𝑒
228,398 227,774 227,846 231,706 232,989 230,945
37,654 37,745 38,375 38,541 38,619 37,980
25 25 25 25 25 25
3 3 3 3 3 3
2,279 2,308 2,960 2,727 2,655 2,327
429 408 411 398 411 417
19.8 19.8 19.8 19.8 19.8 19.8
7.1
Results
Table 1 summarizes the statistical properties and Table 2 the achieved quality scores of the evaluated pipelines including F1 scores combining coverage and precision values for entities and triples. Note that the 𝐽𝑆𝑂𝑁𝑙𝑙𝑚 and 𝑇 𝐸𝑋𝑇𝑙𝑙𝑚 pipelines could only be executed on the smaller 1K version of the benchmark so that comparability of their results is limited. The results show clear differences between source formats and between baseline and LLM-enhanced variants. Among all pipelines, the RDF pipelines achieve the highest overall quality, confirming that structured sources combined with strong matching lead to the most reliable integration. RDF 𝑙𝑙𝑚 achieves the strongest results (𝐹 1𝐸 = 0.973, 𝐹 1𝑇 = 0.901), improving over RDF 𝑏𝑎𝑠𝑒 particularly in fact-level recall. This indicates that improvements in ontology/schema matching propagate to downstream tasks such as entity resolution and fusion, ultimately improving end-to-end KG quality. The task-specific matching results in Appendix (see Table 4) support this observation by showing that RDF𝑙𝑙𝑚 achieves stronger ontology matching quality while maintaining high precision. The JSON pipelines reach a medium quality. JSON 𝑏𝑎𝑠𝑒 achieves competitive entity coverage, but lower fact recall indicating losses during mapping and relation generation. JSON 𝑙𝑙𝑚 is executed on the small benchmark only but is still worse than JSON 𝑏𝑎𝑠𝑒 especially regarding entity coverage and fact precision. It also has the highest duplication rate (DR) and by far the highest execution time. The text pipelines show the weakest performance across both coverage and correctness, with very low entity and triple recall. This directly exposes knowledge extraction and entity linking as the major bottlenecks for unstructured sources. Precision is substantially higher than recall, indicating that extracted assertions
Marvin Hofer and Erhard Rahm
Table 2: Quality metrics for all pipelines after the final integration step, combining entity- and fact-level coverage and correctness, with ontology-based consistency scores. The results highlight clear differences between pipeline types, with RDFbased pipelines achieving the highest overall quality, while JSON and especially text-based pipelines show reduced accuracy and coverage due to mapping and extraction errors. Higher values indicate better quality. * the 𝐽𝑆𝑂𝑁𝑙𝑙𝑚 and 𝑇 𝐸𝑋𝑇𝑙𝑙𝑚 pipelines are only executed for the 1k version of the benchmark. Pipeline
𝐶𝑜𝑣 𝐸 𝐶𝑜𝑟𝑟 𝐸
𝐹 1𝐸
𝐶𝑜𝑣𝑇 𝐶𝑜𝑟𝑟𝑇
𝐹 1𝑇
1 − 𝐷𝑅 𝑂 𝐷𝑇
𝑅𝐷𝐹𝑏𝑎𝑠𝑒 𝑅𝐷𝐹𝑙𝑙𝑚 𝐽𝑆𝑂𝑁𝑏𝑎𝑠𝑒 𝐽𝑆𝑂𝑁𝑙𝑙𝑚 * 𝑇 𝐸𝑋𝑇𝑏𝑎𝑠𝑒 𝑇 𝐸𝑋𝑇𝑙𝑙𝑚 *
0.858 0.959 0.791 0.673 0.020 0.042
0.875 0.988 0.824 0.924 0.285 0.322
0.866 0.973 0.807 0.779 0.037 0.074
0.541 0.884 0.457 0.517 0.009 0.044
0.901 0.917 0.871 0.587 0.191 0.160
0.676 0.901 0.600 0.549 0.017 0.069
0.995 0.995 0.995 0.973 1 1
0.994 0.995 0.988 1 1 1 0.993 0.996 0.989 1 1 1 0.995 0.996 0.842 1 0.794 1 0.987 0.987 0.992 0.992 1 1 0.978 0.969 0.974 0.997 1 0.997 0.737 0.551 0.867 0.849 1 0.939
𝑅𝐽𝑇𝑏𝑎𝑠𝑒 𝑅𝑇 𝐽𝑏𝑎𝑠𝑒 𝐽 𝑅𝑇𝑏𝑎𝑠𝑒 𝐽𝑇 𝑅𝑏𝑎𝑠𝑒 𝑇 𝐽 𝑅𝑏𝑎𝑠𝑒 𝑇 𝑅𝐽𝑏𝑎𝑠𝑒
0.446 0.447 0.444 0.457 0.461 0.453
0.819 0.818 0.782 0.797 0.801 0.817
0.578 0.578 0.566 0.581 0.585 0.583
0.306 0.306 0.306 0.326 0.328 0.318
0.602 0.605 0.600 0.615 0.614 0.611
0.406 0.406 0.406 0.426 0.428 0.418
0.996 0.996 0.995 0.995 0.994 0.996
0.966 0.961 0.959 0.999 0.964 0.957 0.952 0.999 0.97 0.961 0.962 0.999 0.97 0.961 0.958 0.999 0.966 0.958 0.957 0.999 0.964 0.959 0.957 0.999
are often plausible but highly incomplete. Again the LLM variant JSON 𝑙𝑙𝑚 is very slow despite its execution on the small benchmark. In contrast to the RDF pipelines, the LLM-enhanced text pipeline does not consistently improve task-level recall (see Appendix Table 5). Nevertheless, it achieves slightly better end-to-end scores due to producing more precise and semantically consistent extractions, illustrating that improvements in downstream KG quality do not necessarily correlate with higher extraction recall alone. Differences between entity- and triple-level scores also reveal integration problems. For example, higher entity recall than triple recall indicates that entities may be introduced successfully but remain weakly connected or incompletely populated. Conversely, precision degradation together with duplicate penalties points to entity resolution problems, where unresolved duplicates inflate graph size while reducing semantic quality. The ontology-based consistency metrics provide complementary insights not captured by coverage or correctness alone. The RDF pipelines show near-perfect consistency, indicating that structured integration largely preserves ontology constraints. Hence, their strong coverage and correctness are not achieved at the expense of structural correctness. In contrast, JSON and especially text pipelines show lower compliance in domain/range constraints and relation usage. These violations indicate concrete integration defects, such as incorrectly mapped predicates, missing or wrong types, or semantically invalid links. JSON 𝑏𝑎𝑠𝑒 has most violations for relation direction O𝑅 and literal datatypes O𝐿𝑇 as well as the highest number of un-typed entities (Table 1) indicating problems in data mapping to RDF and type inference. For the text pipelines, consistency degradations align with the weak extraction and linking performance seen in the coverage and correctness metrics, showing how early-stage extraction errors propagate into KG consistency problems. The six MSP pipelines perform similarly and reach modest F1 scores especially since there is always a low-quality text pipeline
𝑂𝐷
𝑂𝑅
𝑂 𝑅𝐷
𝑂 𝐿𝑇
1 1 1 1 1 1
𝑂 𝐿𝐹
1 0.999 1 0.999 0.999 0.999
involved. Their man problem is a low coverage while precision values are competitively high, especially for triples. The order in which the source formats are integrated, e.g., whether text data is integrated first or last, seems to have little influence on the final KG quality.
Incremental behavior. Figure 5 shows how the size of the integrated KG evolves over successive source integrations for the three classes Film, Person and Company. RDF pipelines maintain stable increases across stages and can nearly match the size of the reference graphs indicating a continuous high coverage. By contrast, the pure text pipelines can hardly add further entities compared to the seed graph for Person and company entities. The MSP pipelines struggle the most for entities of the most frequent class Person since both JSON and TEXT steps have difficulties with this entity type.
Summary of findings. Overall, the evaluation shows integrating structured (RDF) data has clear advantages over the integration of semi-structured and text data. LLM use was very beneficial for RDF input but very slow and ineffective for JSON and text data. The observed limitations also show that most pipelines and their configurations are far from perfect providing room for improvement. The proposed benchmark metrics proved to be useful to identify integration problems and pipeline weaknesses. Low coverage exposes incomplete extraction or mapping, reduced precision and duplicate penalties expose linking and fusion errors, and consistency violations reveal ontology-level defects. Across all pipelines, entity resolution and schema alignment emerge as major drivers of KG quality, while knowledge extraction remains the principal bottleneck for unstructured data.
Evaluation of Pipelines for Data Integration into Knowledge Graphs
Figure 5: Number of integrated entities by entity type and expected entities at each source increment (stage) for all pipelines. The dotted lines indicate the values of the current reference between the seed (red) and the three source increments (black).
7.2
Comparative rankings
To compare pipelines under different evaluation priorities, we aggregate the three main quality dimensions introduced in Section 3.2, namely coverage, correctness, and consistency. For each dimension, we first compute the arithmetic mean of the corresponding subgroup metrics: entity and triple recall for coverage, entity and triple precision for correctness, and the ontology consistency metrics for consistency. We then enumerate all weight combinations for the three quality scores using a step size of 0.1, resulting in 66 distinct weight permutations. Table 3 summarizes the resulting aggregated subgroup scores together with their harmonic mean and the observed rank ranges across all weight permutations. The results reveal a clear hierarchy of pipeline performance. The RDF 𝑙𝑙𝑚 pipeline consistently achieves the best results across all quality dimensions, resulting in the highest harmonic mean (0.956) and a stable rank range of only 1–1. This indicates that the pipeline remains the top-ranked approach independently of the weighting configuration. The RDF 𝑏𝑎𝑠𝑒 pipeline also performs strongly, achieving the secondhighest harmonic mean (0.843) with a fixed rank range of 2–2, confirming the robustness of RDF-based integration approaches. The JSON pipelines achieve intermediate results. While JSON 𝑏𝑎𝑠𝑒 obtains a slightly higher harmonic mean than JSON 𝑙𝑙𝑚 , its much larger rank range (3–11) indicates a stronger dependence on the weighting configuration. In contrast, JSON 𝑙𝑙𝑚 exhibits more stable rankings despite slightly lower aggregated quality values. The MSP pipelines occupy the middle range of the ranking with harmonic means between approximately 0.585 and 0.604. Their relatively narrow score differences indicate that the integration order has only limited influence on the final KG quality, although moderate rank variability remains depending on the weighting priorities. The text-based pipelines achieve the lowest overall results. In particular, TEXT 𝑏𝑎𝑠𝑒 and TEXT 𝑙𝑙𝑚 obtain very low harmonic means due to their limited coverage and correctness despite comparatively high consistency values. This shows that knowledge extraction and
linking remain the primary bottlenecks for unstructured source integration. Beyond comparing pipelines, the ranking analysis also illustrates an important property of the benchmark: pipeline comparisons remain largely stable under changing user priorities. This indicates that the benchmark does not overfit to a particular weighting choice and that observed quality differences reflect robust pipeline characteristics rather than artifacts of score aggregation. Taken together, the comparative rankings reinforce the conclusions from the metric-level analysis: structured integration pipelines provide the most reliable KG quality, LLM enhancements can improve specific integration stages depending on the source type, and weaknesses in extraction or matching systematically propagate into lower end-to-end rankings.
8
CONCLUSIONS AND OUTLOOK
We presented KGI-Bench, a benchmark for the end-to-end evaluation of knowledge graph integration pipelines. In contrast to prior work that primarily evaluates isolated tasks such as extraction, matching, or ontology alignment, our approach focuses on assessing the quality of the integrated knowledge graph produced by complete pipelines or sequences of several pipelines. KGI-Bench is based on the three complementary metrics coverage, correctness, and consistency to comprehensively evaluate the quality of the integrated KG. Furthermore, we provide the open domain-specific benchmark KGI-Bench-Movie for incremental integration of heterogeneous sources and comparisons with a reference KG. We demonstrated the usefulness of the new benchmarks by a comparative evaluation of baseline and LLM-enhanced pipelines for RDF, JSON, and text sources. The results demonstrate clear differences in integration quality across source types and pipeline designs. Structured RDF pipelines achieved the strongest and most robust results, while semi-structured and especially text-based pipelines revealed substantial challenges in mapping, extraction, and linking. The evaluation further showed that improvements in individual tasks, such as schema matching, can measurably improve end-toend KG quality, while weaknesses in a single stage can dominate
Marvin Hofer and Erhard Rahm
Table 3: Comparative ranking of pipelines based on the aggregated subgroup scores for coverage, accuracy, and consistency. The Harmonic Mean summarizes the three quality dimensions while penalizing imbalanced performance. The Rank Range column shows the observed rank interval across all weight permutations used for the total score calculation, indicating the robustness of pipeline rankings under different evaluation priorities. Pipeline
Cov.
Corr.
Cons.
H-Mean
Rank Range
𝑅𝐷𝐹𝑙𝑙𝑚 𝑅𝐷𝐹𝑏𝑎𝑠𝑒 𝐽𝑆𝑂𝑁𝑏𝑎𝑠𝑒 𝐽𝑆𝑂𝑁𝑙𝑙𝑚 * 𝑇 𝐽 𝑅𝑏𝑎𝑠𝑒 𝐽𝑇 𝑅𝑏𝑎𝑠𝑒 𝑇 𝑅𝐽𝑏𝑎𝑠𝑒 𝑅𝑇 𝐽𝑏𝑎𝑠𝑒 𝑅𝐽𝑇𝑏𝑎𝑠𝑒 𝐽 𝑅𝑇𝑏𝑎𝑠𝑒 𝑇 𝐸𝑋𝑇𝑙𝑙𝑚 * 𝑇 𝐸𝑋𝑇𝑏𝑎𝑠𝑒
0.921 0.7 0.624 0.595 0.395 0.392 0.386 0.377 0.376 0.375 0.043 0.014
0.953 0.888 0.847 0.756 0.708 0.706 0.714 0.712 0.71 0.691 0.241 0.238
0.996 0.996 0.946 0.99 0.982 0.983 0.982 0.981 0.983 0.984 0.849 0.988
0.956 0.843 0.781 0.747 0.604 0.601 0.598 0.59 0.59 0.585 0.105 0.04
1-1 2-2 3 - 11 3-4 5-9 5-8 4-9 4-9 6 - 10 5 - 10 4 - 12 11 - 12
overall pipeline performance. These findings underline the importance of evaluating integration pipelines holistically rather than through isolated task metrics alone. The evaluation also showed that the proposed metrics can serve as diagnostic indicators of characteristic integration problems. Coverage metrics expose incompleteness, correctness metrics reveal matching and fusion errors, and consistency metrics capture structural and semantic defects that would otherwise remain hidden. There are several opportunities for future work. There are many ways to improve pipeline design, including additional LLM-based approaches and more advanced fusion or reasoning techniques, and test these with the existing benchmark. The benchmark can be extended beyond the movie domain to additional domains and ontologies in order to study additional integration challenges and increase the generality of the evaluation methodology. It is furthermore promising to investigate automated support for diagnosing pipeline weaknesses and guiding the improvement of pipeline designs based on the observed evaluation patterns.
ACKNOWLEDGMENTS The authors acknowledge the financial support by the Federal Ministry of Education and Research of Germany and by the Sächsische Staatsministerium für Wissenschaft Kultur und Tourismus in the program Center of Excellence for AI-research "Center for Scalable Data Analytics and Artificial Intelligence Dresden/Leipzig", project identification number: ScaDS.AI.
REFERENCES [1] Gabriel Amaral, Odinaldo Rodrigues, and Elena Simperl. 2024. ProVe: A pipeline for automated provenance verification of knowledge graphs against textual sources. Semantic Web 15, 6 (2024), 2159–2192. https://doi.org/10.3233/SW233467
[2] Renzo Angles, Angela Bonifati, Stefania Dumbrava, George Fletcher, Alastair Green, Jan Hidders, Bei Li, Leonid Libkin, Victor Marsault, Wim Martens, Filip Murlak, Stefan Plantikow, Ognjen Savkovic, Michael Schmidt, Juan Sequeda, Slawek Staworko, Dominik Tomaszuk, Hannes Voigt, Domagoj Vrgoc, Mingxi Wu, and Dusan Zivkovic. 2023. PG-Schema: Schemas for Property Graphs. Proc. ACM Manag. Data 1, 2 (2023), 198:1–198:25. https://doi.org/10.1145/3589778 [3] Roos M Bakker and Maaike HT de Boer. 2026. Dynamic knowledge graph evaluation: Semantic and syntactic metrics for evaluating changes. Data & Knowledge Engineering (2026), 102611. [4] Meghyn Bienvenu and Camille Bourgaux. 2017. Inconsistency-Tolerant Querying of Description Logic Knowledge Bases. In Reasoning Web: Logical Foundation of Knowledge Graph Construction and Query Answering, Jeff Z. Pan, Diego Calvanese, Thomas Eiter, Ian Horrocks, Michael Kifer, Fangzhen Lin, and Yuting Zhao (Eds.). Vol. 9885. Springer International Publishing, Cham, 156–202. https: //doi.org/10.1007/978-3-319-49493-7_5 Series Title: Lecture Notes in Computer Science. [5] Christian Bizer and Andy Seaborne. 2004. D2RQ-treating non-RDF databases as virtual RDF graphs. In Proceedings of the 3rd international semantic web conference (ISWC2004), Vol. 2004. Springer Hiroshima. [6] Martin Brümmer, Milan Dojchinovski, and Sebastian Hellmann. 2016. DBpedia Abstracts: A Large-Scale, Open, Multilingual NLP Training Corpus. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (Eds.). European Language Resources Association (ELRA), Portorož, Slovenia, 3339–3343. https://aclanthology.org/L161532/ [7] Ringwald Celian, Gandon, Fabien, Faron Catherine, Michel Franck, and Abi Akl Hanna. 2025. A systematic review of relation extraction task since the emergence of Transformers. https://doi.org/10.48550/arXiv.2511.03610 arXiv:2511.03610 [cs]. [8] Vassilis Christophides, Vasilis Efthymiou, Themis Palpanas, George Papadakis, and Kostas Stefanidis. 2021. An Overview of End-to-End Entity Resolution for Big Data. ACM Comput. Surv. 53, 6 (2021), 127:1–127:42. https://doi.org/10.1145/ 3418896 [9] Carolina Cortes, Lisa Ehrlinger, Lorena Etcheverry, and Felix Naumann. 2025. Is SHACL Suitable for Data Quality Assessment? CoRR abs/2507.22305 (2025). https://doi.org/10.48550/ARXIV.2507.22305 arXiv:2507.22305 [10] Meiji Cui, Li Li, Zhihong Wang, and Mingyu You. 2017. A Survey on Relation Extraction. In Knowledge Graph and Semantic Computing. Language, Knowledge, and Intelligence - Second China Conference, CCKS 2017, Chengdu, China, August 26-29, 2017, Revised Selected Papers (Communications in Computer and Information Science), Juanzi Li, Ming Zhou, Guilin Qi, Ni Lao, Tong Ruan, and Jianfeng Du (Eds.), Vol. 784. Springer, 50–58. https://doi.org/10.1007/978-981-10-7359-5_6 [11] Xi Deng, Volker Haarslev, and Nematollaah Shiri. 2007. Measuring Inconsistencies in Ontologies. In The Semantic Web: Research and Applications, Enrico Franconi, Michael Kifer, and Wolfgang May (Eds.). Vol. 4519. Springer Berlin Heidelberg, Berlin, Heidelberg, 326–340. https://doi.org/10.1007/978-3-540-72667-8_24 Series Title: Lecture Notes in Computer Science. [12] Jérôme Euzenat, Christian Meilicke, Heiner Stuckenschmidt, Pavel Shvaiko, and Cássia Trojahn. 2011. Ontology alignment evaluation initiative: six years of experience. In Journal on data semantics XV. Springer, 158–192. [13] Juliana Freire, Grace Fan, Benjamin Feuer, Christos Koutras, Yurong Liu, Eduardo Peña, and Eden Wu. 2025. Large Language Models for Data Discovery and Integration: Challenges and Opportunities. IEEE Data Eng. Bull 49, 1 (2025), 3–31. [14] Hamed Babaei Giglou, Jennifer D’Souza, Oliver Karras, and Sören Auer. 2025. OntoAligner: A Comprehensive Modular and Robust Python Toolkit for Ontology Alignment. https://doi.org/10.48550/arXiv.2503.21902 arXiv:2503.21902 [cs]. [15] Nicolas Heist, Sven Hertling, and Heiko Paulheim. 2023. KGrEaT: A Framework to Evaluate Knowledge Graphs via Downstream Tasks. https://doi.org/10.48550/ arXiv.2308.10537 arXiv:2308.10537 [cs]. [16] Sven Hertling and Heiko Paulheim. 2020. The Knowledge Graph Track at OAEI. The Semantic Web 12123 (May 2020), 343–359. https://doi.org/10.1007/978-3030-49461-2_20 [17] Marvin Hofer, Daniel Obraczka, Alieh Saeedi, Hanna Köpcke, and Erhard Rahm. 2024. Construction of Knowledge Graphs: Current State and Challenges. Inf. 15, 8 (2024), 509. https://doi.org/10.3390/INFO15080509 [18] Marvin Hofer and Erhard Rahm. 2025. KGpipe: Generation and Evaluation of Pipelines for Data Integration into Knowledge Graphs. CoRR abs/2511.18364 (2025). https://doi.org/10.48550/arXiv.2511.18364 [19] Aidan Hogan, Eva Blomqvist, Michael Cochez, Claudia d’Amato, Gerard De Melo, Claudio Gutierrez, Sabrina Kirrane, José Emilio Labra Gayo, Roberto Navigli, Sebastian Neumaier, et al. 2021. Knowledge graphs. ACM Computing Surveys (Csur) 54, 4 (2021), 1–37. [20] Elwin Huaman. 2022. Steps to Knowledge Graphs Quality Assessment. https: //doi.org/10.48550/arXiv.2208.07779 arXiv:2208.07779 [cs].
Evaluation of Pipelines for Data Integration into Knowledge Graphs
[21] Elwin Huaman, Amar Tauqeer, and Anna Fensel. 2021. Towards Knowledge Graphs Validation through Weighted Knowledge Sources. In Knowledge Graphs and Semantic Web (KGSWC 2021) (Communications in Computer and Information Science), Vol. 1459. Springer, 45–60. https://doi.org/10.1007/978-3-030-91305-2_4 [22] Pere-Lluís Huguet Cabot, Simone Tedeschi, Axel-Cyrille Ngonga Ngomo, and Roberto Navigli. 2023. REDFM : a Filtered and Multilingual Relation Extraction Dataset. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Toronto, Canada, 4326–4343. https://doi.org/10.18653/v1/2023.acl-long.237 [23] Jan Martin Keil. [n.d.]. ABECTO: Assessing Accuracy and Completeness of RDF Knowledge Graphs. ([n. d.]). [24] Dimitris Kontokostas, Patrick Westphal, Sören Auer, Sebastian Hellmann, Jens Lehmann, Roland Cornelissen, and Amrapali Zaveri. 2014. Test-driven evaluation of linked data quality. In 23rd International World Wide Web Conference, WWW ’14, Seoul, Republic of Korea, April 7-11, 2014, Chin-Wan Chung, Andrei Z. Broder, Kyuseok Shim, and Torsten Suel (Eds.). ACM, 747–758. https://doi.org/10.1145/ 2566486.2568002 [25] Dimitris Kontokostas, Patrick Westphal, Sören Auer, Sebastian Hellmann, Jens Lehmann, Roland Cornelissen, and Amrapali Zaveri. 2014. Test-driven evaluation of linked data quality. In 23rd International World Wide Web Conference, WWW ’14, Seoul, Republic of Korea, April 7-11, 2014, Chin-Wan Chung, Andrei Z. Broder, Kyuseok Shim, and Torsten Suel (Eds.). ACM, 747–758. https://doi.org/10.1145/ 2566486.2568002 [26] Christos Koutras, George Siachamis, Andra Ionescu, Kyriakos Psarakis, Jerry Brons, Marios Fragkoulis, Christoph Lofi, Angela Bonifati, and Asterios Katsifodimos. 2021. Valentine: Evaluating Matching Techniques for Dataset Discovery. In 2021 IEEE 37th International Conference on Data Engineering (ICDE). IEEE, 468–479. [27] Domenico Lembo, Maurizio Lenzerini, Riccardo Rosati, Marco Ruzzi, and Domenico Fabio Savo. [n.d.]. Inconsistency-tolerant Query Answering in Ontology-based Data Access. ([n. d.]). [28] Thomas Lukasiewicz, Enrico Malizia, Maria Vanina Martinez, Cristian Molinaro, Andreas Pieris, and Gerardo I. Simari. 2022. Inconsistency-tolerant query answering for existential rules. Artificial Intelligence 307 (June 2022), 103685. https://doi.org/10.1016/j.artint.2022.103685 [29] Stefano Marchesin and Gianmaria Silvello. 2024. Efficient and Reliable Estimation of Knowledge Graph Accuracy. Proceedings of the VLDB Endowment 17, 9 (May 2024), 2392–2403. https://doi.org/10.14778/3665844.3665865 [30] Jose L. Martinez-Rodriguez, Aidan Hogan, and Ivan Lopez-Arevalo. 2020. Information extraction meets the Semantic Web: A survey. Semantic Web 11, 2 (Feb. 2020), 255–335. https://doi.org/10.3233/SW-180333 [31] Pablo N. Mendes, Hannes Mühleisen, and Christian Bizer. 2012. Sieve: linked data quality assessment and fusion. In Proceedings of the 2012 Joint EDBT/ICDT Workshops, Berlin, Germany, March 30, 2012, Divesh Srivastava and Ismail Ari (Eds.). ACM, 116–123. https://doi.org/10.1145/2320765.2320803 [32] Lars-Peter Meyer, Claus Stadler, Johannes Frey, Norman Radtke, Kurt Junghanns, Roy Meissner, Gordian Dziwis, Kirill Bulert, and Michael Martin. 2023. LLM-assisted Knowledge Graph Engineering: Experiments with ChatGPT. In First Working Conference on Artificial Intelligence Development for a Resilient and Sustainable Tomorrow - AI Tomorrow 2023, Leipzig, Germany, 29-20 June, 2023 (Informatik Aktuell), Christian Zinke-Wehlmann and Julia Friedrich (Eds.). Springer, 103–115. https://doi.org/10.1007/978-3-658-43705-3_8 [33] Nandana Mihindukulasooriya, Sanju Tiwari, Carlos F Enguix, and Kusum Lata. 2023. Text2kgbench: A benchmark for ontology-driven knowledge graph generation from text. In International semantic web conference. Springer, 247–265. [34] Sedir Mohammed, Lisa Ehrlinger, Hazar Harmouch, Felix Naumann, and Divesh Srivastava. 2025. The Five Facets of Data Quality Assessment. SIGMOD Rec. 54, 2 (July 2025), 18–27. https://doi.org/10.1145/3749116.3749120 [35] Prakhar Ojha and Partha P. Talukdar. 2017. KGEval: Accuracy Estimation of Automatically Constructed Knowledge Graphs. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, Martha Palmer, Rebecca Hwa, and Sebastian Riedel (Eds.). Association for Computational Linguistics, 1741–1750. https://doi.org/10.18653/V1/D17-1183 [36] George Papadakis, Leonidas Tsekouras, Emmanouil Thanos, Nikiforos Pittaras, Giovanni Simonini, Dimitrios Skoutas, Paul Isaris, George Giannakopoulos, Themis Palpanas, and Manolis Koubarakis. 2020. JedAI3 : beyond batch, blockingbased Entity Resolution. In Proceedings of the 23rd International Conference on Extending Database Technology, EDBT 2020, Copenhagen, Denmark, March 30 - April 02, 2020, Angela Bonifati, Yongluan Zhou, Marcos Antonio Vaz Salles, Alexander Böhm, Dan Olteanu, George H. L. Fletcher, Arijit Khan, and Bin Yang (Eds.). OpenProceedings.org, 603–606. https://doi.org/10.5441/002/EDBT.2020.74 [37] Heiko Paulheim. 2016. Knowledge graph refinement: A survey of approaches and evaluation methods. Semantic Web 8, 3 (Dec. 2016), 489–508. https://doi. org/10.3233/SW-160218 [38] Umair Qudus, Michael Röder, Muhammad Saleem, and Axel-Cyrille Ngonga Ngomo. [n.d.]. Fact Checking over Knowledge Graphs – A Survey. ([n. d.]).
[39] Kashif Rabbani, Matteo Lissandrini, and Katja Hose. 2022. SHACL and ShEx in the Wild: A Community Survey on Validating Shapes Generation and Adoption. In Companion of The Web Conference 2022, Virtual Event / Lyon, France, April 25 29, 2022, Frédérique Laforest, Raphaël Troncy, Elena Simperl, Deepak Agarwal, Aristides Gionis, Ivan Herman, and Lionel Médini (Eds.). ACM, 260–263. https: //doi.org/10.1145/3487553.3524253 [40] André Gomes Regino and Anderson Rossanez. 2026. A Systematic Literature Review on RDF Triple Generation from Natural Language Texts. (2026). [41] Farzad Shami, Stefano Marchesin, and Gianmaria Silvello. 2026. Benchmarking Large Language Models for Knowledge Graph Validation. In Proceedings 29th International Conference on Extending Database Technology, EDBT 2026, Tampere, Finland, March 24-27, 2026. OpenProceedings.org, 551–565. https://doi.org/10. 48786/EDBT.2026.45 [42] Fabian M. Suchanek, Serge Abiteboul, and Pierre Senellart. 2011. PARIS: Probabilistic Alignment of Relations, Instances, and Schema. Proc. VLDB Endow. 5, 3 (2011), 157–168. https://doi.org/10.14778/2078331.2078332 [43] Zequn Sun, Qingheng Zhang, Wei Hu, Chengming Wang, Muhao Chen, Farahnaz Akrami, and Chengkai Li. 2020. A benchmarking study of embedding-based entity alignment for knowledge graphs. arXiv preprint arXiv:2003.07743 (2020). [44] Gyte Tamasauskaite and Paul Groth. 2023. Defining a Knowledge Graph Development Process Through a Systematic Review. ACM Trans. Softw. Eng. Methodol. 32, 1 (2023), 27:1–27:40. https://doi.org/10.1145/3522586 [45] Dylan Van Assche, Thomas Delva, Gerald Haesendonck, Pieter Heyvaert, Ben De Meester, and Anastasia Dimou. 2023. Declarative RDF graph generation from heterogeneous (semi-)structured data: A systematic literature review. Journal of Web Semantics 75 (Jan. 2023), 100753. https://doi.org/10.1016/j.websem.2022. 100753 [46] Xiangyu Wang, Lyuzhou Chen, Taiyu Ban, Muhammad Usman, Yifeng Guan, Shikang Liu, Tianhao Wu, and Huanhuan Chen. 2021. Knowledge graph quality control: A survey. Fundamental Research 1, 5 (Sept. 2021), 607–626. https: //doi.org/10.1016/j.fmre.2021.09.003 [47] Gerhard Weikum, Xin Luna Dong, Simon Razniewski, Fabian Suchanek, et al. 2021. Machine knowledge: Creation and curation of comprehensive knowledge bases. Foundations and Trends® in Databases 10, 2-4 (2021), 108–490. [48] Amrapali Zaveri, Anisa Rula, Andrea Maurino, Ricardo Pietrobon, Jens Lehmann, and Sören Auer. 2015. Quality assessment for Linked Data: A Survey: A systematic literature review and conceptual framework. Semantic Web 7, 1 (March 2015), 63–93. https://doi.org/10.3233/SW-150175
Marvin Hofer and Erhard Rahm
APPENDIX Tables 4 and 5 report task-specific evaluation results for the matching and linking tasks used in the evaluated pipelines. The taskspecific results indicate that RDF 𝑙𝑙𝑚 achieves substantially higher ontology matching recall than RDF 𝑏𝑎𝑠𝑒 while maintaining high precision. This improves downstream entity matching and fusion because more semantically corresponding relations become available during graph alignment. As a consequence, the pipeline integrates more correct triples and achieves higher end-to-end KG coverage and F1 scores. Table 4: Entity matching and ontology matching scores (precision and recall) across all entity types. Pipeline
inc.
𝐸𝑀𝑝
𝐸𝑀𝑟
𝑂𝑀𝑝
𝑂𝑀𝑟
𝑅𝐷𝐹𝑏𝑎𝑠𝑒 𝑅𝐷𝐹𝑏𝑎𝑠𝑒 𝑅𝐷𝐹𝑏𝑎𝑠𝑒 𝑅𝐷𝐹𝑙𝑙𝑚 𝑅𝐷𝐹𝑙𝑙𝑚 𝑅𝐷𝐹𝑙𝑙𝑚 𝐽𝑆𝑂𝑁𝑏𝑎𝑠𝑒 𝐽𝑆𝑂𝑁𝑏𝑎𝑠𝑒 𝐽𝑆𝑂𝑁𝑏𝑎𝑠𝑒
1 2 3 1 2 3 1 2 3
0.98 0.98 0.98 0.98 0.98 0.98 0.99 0.75 0.66
1 1 1 1 1 1 0.41 0.4 0.4
1 1 1 1 1 1 0.43 0.43 0.43
0.63 0.58 0.63 0.83 0.83 0.75 0.38 0.38 0.38
𝑅__𝑏𝑎𝑠𝑒 𝐽 𝑅𝑇𝑏𝑎𝑠𝑒 𝐽𝑇 𝑅𝑏𝑎𝑠𝑒 𝑇 𝐽 𝑅𝑏𝑎𝑠𝑒 𝑇 𝑅𝐽𝑏𝑎𝑠𝑒
1 2 3 3 2
0.98 0.97 0.97 0.97 0.98
1 1 1 1 1
1 1 1 1 1
0.63 0.58 0.58 0.63 0.58
Table 5: Entity linking recall scores for film entities extracted from text documents. Evaluation is restricted to film entities; person and company entities yield even lower results. Pipeline
inc.
𝐸𝐿𝑟
𝑇 𝐸𝑋𝑇𝑏𝑎𝑠𝑒 𝑇 𝐸𝑋𝑇𝑏𝑎𝑠𝑒 𝑇 𝐸𝑋𝑇𝑏𝑎𝑠𝑒 𝑇 𝐸𝑋𝑇𝑙𝑙𝑚 * 𝑇 𝐸𝑋𝑇𝑙𝑙𝑚 * 𝑇 𝐸𝑋𝑇𝑙𝑙𝑚 *
1 2 3 1 2 3
0.43 0.43 0.45 0.21 0.14 0.16
𝑅𝑇 𝐽𝑏𝑎𝑠𝑒 𝑅𝐽𝑇𝑏𝑎𝑠𝑒 𝐽𝑇 𝑅𝑏𝑎𝑠𝑒 𝐽 𝑅𝑇𝑏𝑎𝑠𝑒 𝑇 __𝑏𝑎𝑠𝑒
2 3 2 3 1
0.43 0.45 0.43 0.45 0.43