Benchmark Engineering as a Design Instrument for Heterogeneous Information Systems Jáchym Bártíka,∗, Alžběta Šrůtkováa , Irena Holubováa a
Department of Software Engineering, Charles University, Malostranské nám. 25, Praha 118 00, Czech Republic
arXiv:2607.07175v1 [cs.DB] 8 Jul 2026
Abstract Contemporary information systems operate in heterogeneous and continuously evolving data environments, where representation choices and structural redesign decisions strongly influence system behavior. Existing benchmarking approaches, however, rely mostly on static datasets and fixed schemas, providing limited support for analyzing architectural trade-offs or guiding evolution in multi-model settings. This paper introduces TransforMMer, a framework for evolution-aware and representation-aware benchmark engineering in heterogeneous information systems. The approach treats benchmark construction as a systematic design process: starting from raw data, inferring structure, refining it conceptually, and generating comparable dataset variants across relational, document, and graph systems. The framework is grounded in a unified representation that enables explicit modeling of schemas and cross-model mappings and supports reproducible transformations across alternative representations. We position benchmarking as a system-design tool for evaluating architectural and representation-level decisions in evolving information systems, rather than as a static comparison of database engines. Through controlled benchmark construction scenarios on real-world datasets, we demonstrate how structural redesign steps—such as embedding, enrichment, and hybrid partitioning—affect observed query costs across systems. The results show that performance differences emerge primarily from the interaction between ∗
Corresponding author. Email addresses: [email protected] (Jáchym Bártík), [email protected] (Alžběta Šrůtková), [email protected] (Irena Holubová)
workload and representation design. By enabling systematic generation of structurally distinct yet semantically aligned dataset variants, the proposed approach connects conceptual data modeling with empirical system evaluation and supports reproducible, evolution-aware analysis of heterogeneous information systems. Keywords: multi-model data management, benchmark engineering, schema evolution, heterogeneous information systems, cross-model transformation, reproducible benchmarking
2
1. Introduction and Motivation The emergence of Big Data has significantly changed the landscape of modern information systems. Contemporary applications increasingly operate over heterogeneous data environments that combine relational, document, key-value, and graph data models. This diversity has led to the development of multi-model database systems (MMDBs), which integrate different data abstractions within a single platform and support a broad spectrum of workloads. While multi-model systems improve flexibility and enable model-specific optimizations, they also introduce substantial complexity. Differences in schema structures, storage strategies, indexing mechanisms, and query languages complicate both system design and evaluation. This challenge becomes even more pronounced in environments where schemas evolve over time, data representations change, and systems integrate multiple storage technologies. As a result, systematic evaluation of heterogeneous information systems remains difficult. Benchmarking has traditionally served as a primary method for evaluating database systems and data processing platforms. Established benchmarks, such as TPC [1] for relational systems, LDBC [2] for graph databases, and YCSB [3] for key-value stores, enable controlled, reproducible experiments. However, these benchmarks typically assume fixed schemas, stable workloads, and isolated data models. Consequently, they do not sufficiently capture the dynamic and heterogeneous nature of contemporary information systems. Even recent multi-model benchmarks remain largely static, domain-specific, and difficult to adapt to evolving datasets or alternative structural representations. In real-world information systems, data and schemas rarely remain static. Two major sources of dynamics shape such systems: conceptual schema changes—such as adding or removing attributes, enriching datasets from additional sources, or reorganizing storage layouts—and changes in application workloads and access patterns, such as new query types, shifted analytical demands, or the integration of new data consumers. Both kinds of change frequently require restructuring the system’s logical representations. Notably, workload-driven adaptations need not involve any modification to the underlying conceptual schema; alternative logical mappings are sufficient to accommodate new access patterns. Such changes often influence performance characteristics and system behavior, yet they are rarely captured 3
within benchmarking methodologies. Evaluation is therefore frequently conducted in an ad-hoc manner, and results are difficult to reproduce when structural conditions change. We argue that benchmarking in heterogeneous information systems should be treated as a systematic engineering process rather than as a fixed dataset specification. Such a process should support controlled schema and data evolution, workload-driven adaptation of logical representations, alternative structural representations of the same conceptual data, and reproducible transformation workflows across multiple database systems. Enabling these capabilities allows benchmarking to reflect realistic system evolution and provides deeper insight into trade-offs between data models, storage strategies, and workloads. To address these challenges, this paper introduces an evolution-aware approach to multi-model benchmarking implemented in the TransforMMer framework. The framework integrates schema inference, unified conceptual representation, cross-model transformation, and versioned dataset generation within a single workflow. Instead of providing a single static benchmark dataset, the proposed approach enables systematic generation of multiple structurally distinct yet semantically aligned dataset variants that can be evaluated across heterogeneous database systems. Unlike traditional benchmarks that compare database technologies under fixed schemas, our approach enables exploration of how structural transformations and representation choices influence observed system behavior. This perspective reflects realistic information systems practice, where schema evolution, data integration, and performance-driven redesign are continuous processes rather than one-time decisions. Research Questions. To guide the design and evaluation of the proposed framework, this work addresses the following research questions: • RQ1: How can benchmark datasets for heterogeneous information systems be systematically generated from real-world data while supporting multiple data models and target platforms? • RQ2: How do alternative structural representations of the same conceptual dataset influence observed performance characteristics across relational, document, and graph database systems?
4
• RQ3: To what extent can evolution-aware benchmark generation reveal system-level trade-offs that remain hidden in traditional static benchmark settings? • RQ4: How can transformation-driven benchmark construction support reproducible evaluation and iterative redesign within the information systems life cycle? While benchmarking is traditionally associated with database system evaluation, its role in information systems engineering is broader. In modern data-intensive information systems, structural design decisions, such as schema restructuring, data integration, and model selection, directly influence system performance, maintainability, and the cost of evolution. We therefore position evolution-aware benchmarking as a design-support instrument for heterogeneous information systems rather than merely a performance comparison tool. By enabling controlled exploration of alternative structural representations and storage strategies, the proposed framework supports architectural decision-making throughout the information systems life cycle. Contributions. This paper makes the following contributions: • We conceptualize evolution-aware benchmarking as an architectural design and evaluation methodology for heterogeneous information systems. • We propose a unified framework that integrates schema inference, conceptual modeling, cross-model transformation, and versioned dataset generation into a reproducible workflow. • We demonstrate how controlled structural transformations enable systematic exploration of performance trade-offs across relational, document, and graph database systems. • We introduce DaRe, a repository of datasets, mappings, and transformation artifacts that supports transparency, reuse, and reproducible experimentation. This work contributes to Information Systems research by positioning benchmarking as a design-support methodology for architectural decisionmaking in heterogeneous data-intensive systems. 5
Outline. The remainder of this paper is structured as follows. Section 2 reviews related work in data sources, generators, and benchmarking. Section 3 introduces the conceptual foundations of the approach. Section 4 presents the design and architecture of the framework. Section 5 describes the experimental evaluation. Section 6 discusses implications for information systems research and practice. Section 7 concludes the paper and outlines directions for future work.
6
2. Related Work and Positioning Benchmarking and synthetic data generation are central to evaluating data management technologies and information systems. Existing approaches span traditional single-model benchmarks, synthetic data generators, and more recent multi-model benchmarking initiatives. However, most existing solutions assume static schemas, predefined workloads, and fixed data representations. This section reviews existing approaches and positions the proposed framework within this landscape. 2.1. Traditional Benchmarks and Data Generators Benchmarking has long been used to evaluate the performance and scalability of database systems. Established benchmarks such as TPC-H and TPC-DS provide standardized relational workloads and large-scale data generators for analytical processing. Similarly, YCSB focuses on key-value and cloud-serving systems, while LDBC provides graph workloads for social network scenarios. XML-oriented benchmarks such as XMark [4] and RDF benchmarks such as LUBM [5] support evaluation of semi-structured and semantic data systems. These benchmarks offer well-defined schemas and reproducible workloads, which make them valuable for controlled experiments. However, they typically assume static schemas and a single dominant data model. As a result, they provide limited support for evaluating heterogeneous information systems in which multiple data models coexist and evolve over time. Synthetic data generators provide additional flexibility by enabling the creation of datasets with configurable size and structure. Tools such as SDV [6], Faker [7], and domain-specific generators support the generation of relational, document, or graph data. Graph-oriented generators such as gMark [8] enable configurable graph structures and workloads. Despite their flexibility, most generators focus on individual data models and do not provide unified support for heterogeneous multi-model environments. 2.2. Multi-Model Benchmarks The increasing adoption of heterogeneous data management platforms has motivated the development of multi-model benchmarks. BigBench [9] integrates structured, semi-structured, and unstructured data within a unified benchmark scenario. UniBench [10] and M2Bench [11] extend this direction by supporting multiple data models, including relational, graph, document, 7
and key-value representations. MMSBench [12] focuses on network monitoring scenarios and heterogeneous data ingestion. These approaches represent an important progress toward heterogeneous benchmarking. Nevertheless, most existing multi-model benchmarks remain tied to predefined schemas and specific application domains. They typically provide limited support for systematic schema evolution, controlled structural transformations, or the generation of multiple representation variants of the same dataset. Consequently, they are less suitable for studying how structural redesign and representation choices influence system behavior over time. 2.3. Research Gap Contemporary information systems rarely operate under fixed schemas or static data representations. Instead, they evolve continuously in response to changing workloads, integration requirements, and performance constraints. Structural redesign steps such as embedding, normalization changes, enrichment from additional data sources, or distribution across multiple storage technologies are common in practice. However, existing benchmarking approaches offer limited support for systematically evaluating such evolution. In particular, three limitations can be identified in current benchmarking and data generation approaches: • Limited support for evolution. Most benchmarks assume static schemas and do not support controlled generation of dataset versions reflecting structural change. • Lack of representation variability. Existing benchmarks typically provide a single structural representation of the data, limiting exploration of alternative designs. • Insufficient reproducibility of transformations. Structural modifications applied during system redesign are often ad hoc and difficult to reproduce in benchmarking scenarios. • Limited support for workload-driven adaptation. Existing approaches do not facilitate systematic exploration of alternative logical representations driven by changing access patterns or query workloads. These limitations highlight the need for benchmarking approaches that treat dataset construction as a configurable, reproducible process rather than a fixed specification. 8
2.4. Positioning of the Proposed Approach The framework presented in this paper addresses the above limitations by enabling transformation-driven generation of benchmark datasets across heterogeneous data models. Instead of focusing on predefined schemas or domain-specific workloads, the proposed approach supports systematic construction of multiple structural representations of the same conceptual dataset and controlled propagation of structural evolution. Table 1 summarizes the key differences between existing approaches and the proposed framework. Table 1: Comparison of existing benchmarking and data generation approaches with the proposed framework
Approach
Multimodel support
Schema evolution support
Cross-model transformations
Generalpurpose
TPC-H/TPC-DS XMark [4] LUBM [5] BigBench [9] UniBench [10] M2Bench [11] MMSBench [12] Data generators Graph generators
No No No Partial Yes Yes Yes Partial No
No No Limited No No Limited Limited No Limited
No No No Limited Limited Limited Limited No No
Yes (relational) XML-specific RDF-specific Domain-specific Domain-specific Domain-specific Domain-specific Model-specific Model-specific
Proposed framework
Yes
Yes
Yes
Yes
Unlike existing approaches, the proposed framework treats benchmark construction as a transformation-driven process that supports multiple data models, explicit schema evolution, and reproducible generation of structurally distinct dataset variants. This positioning aligns benchmarking with the realities of contemporary heterogeneous information systems, where structural redesign and incremental evolution are integral parts of system development and operation.
9
3. Conceptual Foundations A central challenge in multi-model data management is the consistent representation and transformation of heterogeneous schemas. Modern database systems support relational tables, nested documents, graphs, and key-value structures, yet structural interoperability across these models remains limited. Our benchmarking approach addresses this limitation by introducing a unified structural representation that enables controlled transformations across models and supports reproducible evaluation. The proposed framework builds on category-theoretic principles and introduces three core constructs: (i) the schema category, (ii) the instance category, and (iii) explicit mappings between system-specific representations and the unified structure. A detailed formal treatment is provided in [13]; here we summarize only the concepts necessary for understanding benchmark construction. 3.1. Unified Structural Representation To reason consistently across heterogeneous models, we introduce modelagnostic terminology: • Entity – a stored entity (e.g., tuple, document, node), • Property – an attribute of an entity, • Reference – a relationship between entities. These abstractions allow structural reasoning independent of implementation syntax. Category theory provides the formal basis for this unification. A category consists of objects and morphisms between them, together with composition and identity operations. In our context, objects correspond to structural elements (e.g., entity types), while morphisms represent relationships or dependencies. This abstraction allows heterogeneous schemas to be represented within a single canonical structure. Running Example. Consider a simple dataset describing Users and Friends. Each user also has an Address. In a relational database system, this information can be represented using three tables: • user(id, name) 10
• friend(user_id, friend_id, since) • address(user_id, street, city, state) Here, the friend table implements a many-to-many relationship between users and their friends through foreign keys. In a document-oriented system, the same conceptual information may be represented differently—we might have a single user collection where each document contains an embedded array of friends and an embedded address subdocument. In a graph database, user and address would be represented as node labels, while FRIEND and USER_ADDRESS relationships would connect the nodes to each other. Although these representations differ structurally, they capture the same conceptual semantics. Within the schema category, User, Friend, and Address, along with their properties, are modeled as objects, while links between them are represented as morphisms. Concrete design choices—such as join tables, document embeddings, or graph edges—correspond to alternative mapping configurations defined over the same canonical schema. Friend { 7.1, 8.1 }
Street City
6
4
Address { 3.1 }
7 3
5
8
User { 1 }
1
Since
9
Id
2
State
Name
Figure 1: Example schema category for the user-friend-address model.
The example schema category is depicted in Figure 1. Note the numbers next to morphisms, which are called signatures and whose purpose is to uniquely identify morphisms. Next, the numbers in curly braces represent identifiers of objects. For example, the User object is identified by the property accessible through the morphism with signature 1, i.e., the Id property. Address is identified by a composite morphism 3.1 which ultimately leads to the Id property. Lastly, Friend is identified by a composite key of two user identifiers. 3.2. Schema and Instance Categories The schema category captures structural definitions independent of a particular database system. Relational tables, document collections, or graph 11
labels are treated uniformly as objects within the same structure. Structural differences—such as normalization, embedding, or edge projection—are expressed through morphism configurations. The instance category mirrors the schema category but contains concrete data. An example of such structure is shown in Figure 2. There are five object records, each represented as a tuple (signature, value). These records are divided among domains of three instance objects (User, Friend, and Address). There are also four morphism records, represented as tuples consisting of two object records. Each of the morphism records belongs to the domain of one of the three morphisms (3, 7, and 8). Address
3
User
7
3.1: "u_1", 4: "123 Main St", 5: "New York", 6: "NY",
1: "u_1", 2: "Alice"
3.1: "u_2", ...
1: "u_2", 2: "Bob"
8
Friend
7.1: "u_1", 8.1: "u_2", 9: "2026",
Figure 2: Example instance category for the schema category in Figure 1. The dashed lines separate the domains of the instance objects/morphisms.
3.3. Mappings and Structural Transformations Mappings connect system-specific schemas with the unified schema category. They define how kinds (relational tables, document collections, or graph nodes and relationships) correspond to canonical objects and morphisms. For instance, a relational join table mapping would specify how foreign keys relate to morphisms in the schema, while a document embedding mapping would indicate which properties are nested within others. In the running example, choosing a relational join table, document embedding, or graph edge representation corresponds to defining different mappings over the same conceptual schema. Each mapping yields a structurally distinct dataset while preserving logical semantics. Examples of such mappings are shown in Figure 3. Each mapping consists of a JSON-like structure where attribute names correspond to properties of the given kind and values correspond to signatures of morphisms in the schema category. For example, object User is mapped to a relational table 12
user: { id: 1, name: 2, }
user
id
street
city
state
u_1 123 Main St New York NY u_2
...
id
name
u_1 Alice u_2
Bob
...
...
FRIEND: { _from.User: 7 { id: 1, }, _to.User: 8 { id: 1, }, since: 9, }
user: { _id: 1, address: -3 { street: 4, city: 5, state: 6, }, friends: -7 { id: 8.1, since: 9, } }
user
address: { id: 3.1, street: 4, city: 5, state: 6, }
address
user with attributes id and name corresponding to morphisms with signatures 1 and 2, respectively. The same object is also mapped to a document collection user which, however, contains some additional attributes (nested document address and an array of nested documents friends). Relational model cannot contain nested attributes, so Address has to be mapped to a separate table. [ { _id: "u_1", name: "Alice", address: { street: "123 Main St", city: "New York", state: "NY", }, friends: [ { id: "u_2", since: "2026" } ], }, { _id: "u_2", name: "Bob", address: { street: "...", city: "...", state: "...", }, friends: [ ], }]
FRIEND User id: u_2
since: 2026
User id: u_1
Figure 3: Example mapping from the schema category in Figure 1 to various kinds in relational (purple), graph (blue), and document (green) models.
This explicit mapping mechanism enables controlled structural variation. For benchmarking, this is critical: multiple structural realizations of the same dataset can be generated and evaluated under identical logical semantics. Structural evolution can be interpreted as a sequence of mapping modifications applied to the schema category and propagated to the instance category. 3.4. Formalization Formally, a schema category S is a category S = (OS , MS , ◦), where OS is a set of objects, MS is a set of morphisms between the objects, and ◦ is the composition operation. Similarly, we can define an instance category I as a category of instance objects and instance morphisms that mirror the structure of the schema category and contain concrete data. An instance is then a functor inst : 13
S → I, mapping schema category to instance category while preserving the structure defined by the morphisms. Finally, a mapping from the schema category S to a system-specific logical representation C is defined as a functor M : S → C, where C is the category representing the system-specific schema. The functor preserves the structure of the schema category while translating it into the target representation. We use category theory as a formal foundation for our framework, but the practical implementation abstracts away from these details. This allows us to focus on the engineering aspects of benchmark construction while ensuring a rigorous theoretical basis for structural representation and transformation. 3.5. Implications for Benchmark Construction In this work, we do not claim full semantic equivalence in the sense of arbitrary query equivalence across different data models. Instead, we use the term semantic alignment to denote that generated variants are derived from the same canonical schema category and instance category, preserve the same selected entity identities, properties, and references, and are produced by explicit mappings recorded as part of the benchmark configuration. Thus, two variants may differ structurally, for example by normalization, embedding, or graph projection, but their corresponding elements remain traceable to the same canonical objects and morphisms. This notion is sufficient for the benchmark engineering objective of this paper: evaluating how alternative representations of the same conceptual data affect selected workloads. The combination of unified schema representation, instance-level alignment, and explicit mappings enables configurable benchmark engineering. Instead of evaluating systems using a single fixed schema, multiple structural variants and evolution scenarios can be systematically generated and reproduced. This capability is particularly important in heterogeneous information systems, where data models coexist, and structural redesign is common. By grounding benchmark construction in a unified abstraction, the framework allows rigorous evaluation of representation choices and transformation strategies across evolving multi-model environments. The following section describes how these conceptual foundations are operationalized within the proposed benchmarking framework.
14
4. Design of the Evolution-Aware Benchmarking Framework Based on the conceptual foundations introduced in the previous section, we now present the design of an evolution-aware benchmarking framework for heterogeneous information systems and its implementation in the TransforMMer platform. The framework operationalizes unified multi-model schema representation and transformation-driven dataset generation, enabling systematic construction of benchmark datasets across heterogeneous data models. While TransforMMer has been previously introduced in a vision paper [14] and a demonstration paper [15], those works focused primarily on conceptual motivation and prototype demonstration. In contrast, this paper positions the framework as an infrastructure for reproducible and evolutionaware benchmarking and provides a detailed description of its design principles, workflow, and architectural organization. 4.1. Design Rationale The framework is designed to address three core requirements of benchmarking in heterogeneous information systems: • Unified structural representation: Heterogeneous datasets originating from relational, document, or graph models must be represented in a model-independent manner. • Controlled evolution: Benchmark datasets should support structural redesign and schema evolution scenarios that reflect real-world system changes. This encompasses both (i) conceptual schema changes propagated consistently to all logical representations and queries, and (ii) workload-driven adaptations of logical representations that preserve the underlying conceptual schema. • Reproducibility: Alternative structural variants must be generated in a controlled and repeatable way to enable systematic comparison. These requirements motivated the separation of the benchmarking process into three stages: schema inference, unified schema construction, and transformation-driven dataset generation. This separation allows structural modifications to be introduced at a conceptual level and propagated consistently across multiple target representations. 15
4.2. Benchmark Generation Workflow The framework follows a structured pipeline that transforms heterogeneous input data into reproducible multi-model benchmark datasets. The workflow consists of three main stages: 1. Schema inference from raw input data, 2. Unified schema construction and refinement, 3. Data transformation and generation for selected target systems.
2
schema category
input mappings
MM-evocat
3
schema category
input mappings
output mappings
output model selection
identifier and reference candidates
modifications
Inputs SQL, CSV, JSON, ...
inference
Figure 4 illustrates the overall workflow.
data
1
Output
MM-infer
Figure 4: Benchmark generation workflow from heterogeneous input data to multi-model benchmark outputs. Adapted from [15]. See Figure 5 for a concrete example of the workflow stages.
To illustrate the workflow, we continue with the dataset describing Users and Friends (see Figure 5). Assume the input is provided in a semi-structured format (e.g., JSON), where user records contain nested addresses. Another input (e.g., CSV) then contains the friendships. The framework first performs schema inference, extracting structural elements such as entities (i.e., User, Address, and Friend), attributes (e.g., Id), reference candidates (e.g., from From to Id), identifier candidates (e.g., User → Id), and mappings between the input data models and the inferred schema category components. By accepting the candidates, the user merges the schema category components into one and defines object identifiers. Candidates can also be rejected, or created manually. The refined schema is displayed in Figure 1. At this level, the conceptual relationship between users, friends, and addresses is represented independently of whether it will later be realized as a join table, 16
from
to
since
u_1
u_2
2026
Friend
6
3
8
User { 1 }
state
u_1 123 Main St New York NY u_2
...
...
...
id user
address
User
city
Id
2
Name
name
u_1 Alice u_2
Reference candidates
From - Id, To - Id,
Id 2
user: { id: 1, name: 2, }
id: u_1 street
1
Name
3
since: 2026
1
User
Since
9
2
FRIEND
id
3
5
State
id: u_2
4
Address
State
5
User
6
City
7
Address { 3.1 }
Since
Street
Friend { 7.1, 8.1 } 4
9
8
To
1
Street City
7
From
[ { id: "u_1", name: "Alice", address: { street: "123 Main St", city: "New York", state: "NY" }, }, { id: "u_2", name: "Bob", address: { street: "...", city: "...", state: "..." }, }]
user
friend
an embedded collection, or a graph edge. In the background, the framework updates the inferred mappings to reflect the changes to the schema category. We can use these mappings to create an instance category that represents the data in a model-independent way (see Figure 2).
address: { id: 3.1, street: 4, city: 5, state: 6, }
Identifier candidates
User: { 1 }, Address: { 3.1 }, Friend: { 7, 8 },
FRIEND: { _from.User: 7 { id: 1, }, _to.User: 8 { id: 1, }, since: 9, }
Bob
Figure 5: Inputs (JSON and CSV data), intermediate artifacts (schema category components, reference and identifier candidates), and outputs (unified schema, relational and graph data) of the user-friend-address example. The step numbers correspond to the workflow stages from Figure 4.
Finally, alternative structural mappings are generated for relational, document, or graph database systems. Each mapping yields a structurally distinct yet semantically aligned dataset variant derived from the same conceptual schema and traceable to the same canonical entities, properties, and references. In this example, the user decides to map the User and Address entities to relational tables, while representing the Friend entity as graph 17
edges. These mappings are then used to export data from the instance category into the target systems. The workflow produces reusable intermediate artifacts, including inferred schema descriptions, unified schema representations, and explicit transformation mappings. These artifacts enable reproducible modification and regeneration of benchmark configurations under alternative representation strategies. Schema Inference. The first stage derives structural information from input datasets. Input data may originate from heterogeneous sources such as CSV exports, JSON documents, or relational dumps. The framework extracts entities, attributes, nested structures, and references and gathers statistical characteristics, including attribute frequencies and candidate keys. Inference functionality is implemented using the MM-infer component [16], which performs distributed processing based on Apache Spark [17]. Local structural descriptions are merged into a global representation that serves as the basis for unified schema construction. Unified Schema Construction. The inferred structural representation is transformed into a unified schema category that captures entities and relationships independently of any specific data model. This representation serves as a stable structural reference for defining benchmark configurations and exploring alternative mappings. An important motivation for constructing the unified schema is that genuinely multi-model datasets are rarely available in practice. Most real-world data is accessible in a single representation (e.g., CSV, JSON, or relational exports), and heterogeneous multi-model versions must be constructed manually. By transforming inferred single-model structures into a unified schema, the framework establishes a conceptual representation from which multiple model-specific variants can be systematically generated. In this way, the unified schema acts as a bridge between single-model input data and reproducible multi-model benchmark datasets. At this stage, structural refinements may be introduced. These refinements correspond to realistic redesign operations observed in evolving information systems, including: • normalization or denormalization, • attribute embedding, 18
• schema enrichment through integration of additional data sources, • partitioning of entities across models. Such refinements can be applied either immediately after schema inference or in later iterations of the workflow. Once the unified schema has been established, subsequent benchmark variants can be generated by modifying mappings and structural configurations rather than re-inferring structure from raw data. This enables simulation of schema and system evolution over time while preserving comparability across benchmark versions. By introducing refinements at the conceptual level, the framework ensures that alternative target representations remain comparable while reflecting controlled structural variation. Data Transformation and Generation. In the final stage, refined schemas and mappings are used to generate datasets for selected database systems. Depending on the configuration, the framework produces relational tables, document collections, graph structures, or hybrid multi-model outputs. Transformations are executed using the MM-evocat framework [18], which manages mapping execution and schema evolution operations. The generated datasets represent alternative structural realizations of the same conceptual data and serve as benchmark inputs for comparative evaluation. 4.3. Framework Architecture The framework’s architecture mirrors the three-stage workflow and comprises corresponding layers: inference, unified schema construction, and transformation (see Figure 4). The inference layer extracts structural information from heterogeneous input sources. The schema construction layer builds and refines the unified schema representation. The transformation layer generates target-specific datasets and manages versioned benchmark outputs. Intermediate artifacts are preserved to ensure reproducibility and facilitate controlled reconfiguration. This layered architecture supports modular extensibility while maintaining a coherent transformation pipeline. 4.4. Benchmark Engineering Capabilities The framework enables systematic construction of benchmark datasets that reflect alternative structural representations and evolution scenarios. Since genuinely heterogeneous multi-model datasets are rarely available in practice, the workflow starts from single-model inputs and derives a unified 19
conceptual schema from which multiple representation variants can be generated. Instead of relying on a single static dataset, the framework produces a family of benchmark variants derived from the same conceptual data through controlled transformations and mapping configurations. Dynamic Benchmarks. Dynamic benchmarks reflect structural or content changes over time and allow evaluation under evolving conditions. After the initial schema inference and unified schema construction, subsequent benchmark versions can be generated by modifying mappings and applying structural refinements, rather than reprocessing raw data. The framework supports dynamic benchmarking through versioned dataset generation and controlled, conceptual-level schema transformations. Crucially, structural modifications may also be introduced exclusively at the logical representation level—by modifying mappings while keeping the schema category fixed— thereby enabling systematic exploration of workload-driven adaptation scenarios in which changing access patterns, rather than schema modifications, motivate structural redesign. Real-World Dataset Evolution. When real-world datasets evolve, updated versions can be processed through the same pipeline while preserving transformation configurations and mappings. Because the unified schema serves as a stable structural reference, new dataset versions can be aligned with existing benchmark variants. This ensures comparability across dataset versions and supports longitudinal evaluation of architectural decisions under changing data conditions. Simulated Schema Evolution. If only static data is available, evolution scenarios can be simulated directly at the unified schema level. Structural refinements or alternative mappings can be introduced without repeating schema inference, enabling controlled creation of benchmark variants that represent alternative structural configurations. In this way, the framework supports experimentation with redesign operations such as embedding, partitioning, or enrichment, even in the absence of naturally evolving data. To support reproducibility, generated datasets are stored together with mappings, transformation specifications, and version metadata describing the applied refinements. This enables systematic comparison of benchmark configurations and repeatable experimental evaluation across iterations. By treating benchmark construction as a configurable, versioned engineering process rather than a fixed dataset design task, the framework supports 20
the systematic evaluation of evolving heterogeneous information systems and enables reproducible exploration of representation-level and architectural design decisions.
21
5. Evaluation through Evolution Scenarios in Information Systems This section evaluates the proposed evolution-aware benchmark engineering approach implemented in TransforMMer. Rather than evaluating database engines in isolation, we interpret each experiment as an information systems design scenario. The objective is to analyze how alternative architectural and representation decisions shape system behavior under comparable workloads and data conditions. Consequently, the experiments are not intended to establish a general ranking of database technologies. Instead, we use controlled benchmark scenarios to demonstrate that (i) comparable benchmark variants can be generated across heterogeneous data models, and (ii) representation choices and evolution-like redesign steps (e.g., embedding, enrichment, partitioning) lead to measurable and explainable differences in observed query costs. From an Information Systems perspective, the scenarios should be interpreted as controlled design experiments rather than database benchmarks. They illustrate how architectural and representation decisions influence the behavior of evolving heterogeneous systems. The experiments are organized as benchmark construction scenarios. Each scenario starts from the same underlying data and applies a specific transformation or representation decision. This mirrors the practice of realistic information systems, where data representations are incrementally adapted in response to evolving requirements and workload pressure. 5.1. Inputs The datasets were selected to be representative of real-world data and to cover a range of structural characteristics relevant to heterogeneous benchmarking. First, publicly available datasets are most commonly distributed as CSV or JSON. We therefore assume that typical TransforMMer users start from these formats, and we adapted our input selection accordingly. Second, we prefer datasets that contain non-trivial structure, including nested objects and references. This supports meaningful transformations and enables workloads reflecting realistic access patterns (joins, multi-hop traversals, aggregation). Finally, we also considered well-known or easy-to-understand datasets, which support clarity and reproducibility.
22
Figure 6: Schema category of the Yelp dataset inferred by TransforMMer.
Yelp Dataset. The Yelp dataset1 is widely used as it provides well-structured real-world data for analytical tasks. It includes businesses, reviews, users, check-ins, and attributes, thus capturing consumer interactions and feedback. The data is provided in JSON format, with each file containing a different type (e.g., business.json, review.json, user.json). The subset used in our experiments is approximately 28 MB. Figure 6 shows the inferred schema category. It contains five kinds corresponding to the five input files: business.json, user.json, review.json, tip.json, and checkin.json. The inferred schema category exhibits a complex structure with many attributes and links between kinds, which makes it suitable for studying alternative representations and their impact on query execution. 1
https://business.yelp.com/data/resources/open-dataset/
23
Figure 7: Schema category of the BibleData dataset inferred by TransforMMer.
BibleData Dataset. The BibleData dataset2 is a large and complex collection of structured information about Bible texts, translations, and metadata in CSV format. It includes persons, relationships, verses, labels, and events, with rich interconnections. This structure is reflected in the inferred schema category in Figure 7. This dataset is not used as an example of a rapidly evolving application domain. It is used as a structurally rich, relationship-centric dataset that allows us to test graph-oriented and hybrid representations under controlled conditions. The full dataset contains 18 files; to keep the experiments manageable, we used 9 files (9 types), totaling approximately 26 MB. 2
https://www.kaggle.com/datasets/bradystephenson/bibledata
24
Figure 8: Schema category of the US Cities data inferred by TransforMMer.
US Cities Data. Finally, we used a simple US cities dataset3 (nearly 3,000 KB) provided as CSV. This dataset is used later to enrich the Yelp dataset. Figure 8 shows its schema category. 5.2. Experimental Setup The experiments explore scenarios that combine (i) structural transformations, (ii) alternative representation choices, and (iii) query workloads executed over the generated datasets. The purpose is to demonstrate that TransforMMer can generate comparable benchmark datasets across models and that representation choices—including evolution-like redesign steps such as embedding or partitioning—have a measurable impact on observed query costs. Queries. All benchmark queries were manually created and translated into the native query languages of the target systems. While the current version of TransforMMer focuses on data transformation, we plan to extend the approach to support the transformation of datasets and their associated queries. In this direction, our team is actively working on a generic query language and internal representation logic [19]. Measurements. Query performance measurements were conducted as follows: • Database systems were executed in Docker containers. 3
https://simplemaps.com/data/us-cities
25
• Only the relevant database container was running during measurements. • Before each set of measurements, the container was restarted to reduce caching effects. • Query execution was automated using Python scripts. • Each query was executed 60 times; the last 50 runs were used to compute the average execution time. The first 10 runs were excluded as a warm-up phase. • We ensured that each database system contained exactly the same dataset. This was automatically enforced by TransforMMer.4 Semantic Alignment Checks. To substantiate comparability of the generated variants, we checked semantic alignment at the level of the benchmark configuration. The checks included preservation of object identifiers, cardinalities of mapped object sets, reference consistency, and agreement of selected validation queries formulated over the canonical schema and translated to the target representation. These checks do not establish general semantic equivalence between database models; rather, they document that the generated variants are comparable for the explicitly defined benchmark workload. Technical Details. Hardware specifications: • CPU: Intel(R) Core(TM) i7-8565U CPU @ 1.80GHz, 4 cores / 8 threads • RAM: 16 GB DDR4 @ 2400 MHz • Operating System: Windows 11, Version 24H2 (OS Build 26100.3775) • Execution Environment: Native execution on a local machine; database systems were run in Docker containers. Software environment: • Docker Version: Docker Version 24.0.6 4
Different systems may store and access data differently; such differences are inherent and may influence performance.
26
• Python Version: Python 3.12.2 Database systems: • MongoDB – Name and Version: MongoDB 8.0.8 – Docker Image: mongo:8.0.8 • PostgreSQL – Name and Version: PostgreSQL 17.4 – Docker Image: postgres:17.4 • Neo4j – Name and Version: Neo4j 2025.03.0 – Docker Image: neo4j:5.15 Setup Limitations and Interpretation. The setup was controlled to keep the measurements comparable between systems. Nevertheless, certain limitations can influence the stability. The experiments were run on a personal Windows machine using Docker containers. In Windows, Docker typically relies on a virtualized Linux environment, which can introduce overhead and variability in resource scheduling compared to native Linux or dedicated servers [20]. Desktop background processes, thermal throttling, or variable CPU clock speeds can also contribute to noise. To reduce these effects, containers were restarted before measurements, no other workloads were active, and we used a warm-up phase. Figure 9 illustrates the stabilization behavior over 60 repetitions for the first five queries of the Yelp experiment on PostgreSQL (Section 5.3.1). The first runs show elevated execution times; later runs stabilize. We excluded the first 10 runs for robustness between experiments. Therefore, the reported results are internally consistent and suitable for comparing relative trends caused by structural transformations. However, absolute execution times should not be interpreted as definitive deployment benchmarks. The primary objective is to observe how alternative representations and evolution-like redesign steps influence performance within a reproducible experimental setting. 27
Figure 9: Ramp-up curves for the first five queries (Q1–Q5) executed on PostgreSQL with the Yelp dataset.
5.3. Benchmark Construction Scenarios This subsection presents benchmark scenarios constructed using TransforMMer. Each scenario starts from the same underlying data but applies a controlled transformation or representation decision. In this way, the experiments emulate typical evolution steps in information systems, such as denormalization/normalization, embedding, enrichment, or hybrid partitioning. 5.3.1. Yelp: Query Performance under Alternative Representations We start with the Yelp dataset and evaluate how different representations of the same data influence query costs in relational and document settings. Table 2 lists the query set. Although most queries were created for this work, Q2 and Q3 are inspired by existing studies to maintain comparability and realism. Query Q2 was inspired by Zhang [21], where Yelp data is loaded into PostgreSQL [22] and explored using joins and aggregations. Query Q3 is inspired by both Zhang [21] and Alam et al. [23]. The remaining queries cover patterns from filtering and sorting to join-intensive and multi-hop style access.
28
Table 2: Experimental Queries for Yelp dataset Query Q1
Q2
Q3
Q4
Q5 Q6
Q7
Query Description
Type
Inspired by
Note
Find top 10 businesses in New Orleans with rating ≥ 4.5 and more than 100 reviews. Find users who have reviewed at least 2 businesses in the same city. Count how many businesses saw a drop in average rating from 2019 to 2021. For a given user, find all other users who reviewed the same businesses. Find the average star rating per state per year. For each city, find the user with the highest average review stars. Find businesses frequently reviewed by the same group of users (≥3).
Filter + Sort
New
Classic benchmark
Group Join
+
Zhang 2014 [21]
Temporal Aggregation Join
Zhang 2014 [21], Alam et al. 2021 [23] New
User preference patterns Rating trend signal
Aggregation
New
Group Max
New
Pattern
+
New
Multi-hop style query Time-series aggregation Grouping and ranking Join-heavy workload
Scenario A: Alternative Persistence Architectures for the Same Information System. This scenario represents an architectural decision between alternative persistence strategies for the same information system. We first construct two baseline representations: a document-oriented representation in MongoDB [24] and a relational representation in PostgreSQL. Figure 10 summarizes the transformation. In both cases, we transform the full schema category. For PostgreSQL, kinds with complex properties require decomposition into relationally manageable tables; this is illustrated for Business, where nested attributes are separated. We execute the query set (Table 2) in the native query languages and compare average execution times. Figures 11 and 12 show the results (linear and log scale). The baseline scenario shows that the same analytical intent can yield different computational costs depending on the representation choices and query mechanisms. In particular, join-like patterns implemented via $lookup stages tend to be more expensive in this configuration, while PostgreSQL executes join-heavy aggregations using more mature join and 29
Review Business
User Tip
Checkin
{;} JSON
attribute
Review
Business
Checkin
Review User
Business
Tip
Checkin
User Tip
{;}
Figure 10: Transformations of the Yelp dataset to MongoDB and PostgreSQL
30
Figure 11: Average execution times of the queries from Table 2 on the Yelp dataset, comparing MongoDB and PostgreSQL. Error bars represent one standard deviation over the last 50 runs per query.
aggregation strategies. The two inspired queries (Q2, Q3) support comparability with previous work. Zhang [21] uses similar join-heavy exploration in PostgreSQL [22]. Alam et al. [23] emphasize year-over-year trends. Our scenario extends these analyzes by quantifying the cost of comparable analytical patterns across alternative representations. We also note that query Q4 required a preparatory step in MongoDB (precomputing the list of reviewed businesses), which was not included in the measured execution time and may underestimate the MongoDB cost. In PostgreSQL, the logic is executed within a single query. Finally, while PostgreSQL outperforms MongoDB in these join-heavy workloads, MongoDB can outperform relational systems in other scenarios, particularly at scale and under horizontal execution strategies, as discussed by Makris et al. [25]. Scenario B: Representation-Level Redesign Driven by Workload Requirements. This scenario reflects workload-driven structural evolution within a document-oriented information system. We emulate a typical evolution step in document-oriented information systems: embedding selected data to re31
Figure 12: Average execution times of the queries from Table 2 on the Yelp dataset, comparing MongoDB and PostgreSQL. Error bars represent one standard deviation over the last 50 runs per query. Time is shown on a logarithmic scale.
duce join-like operations. Based on the baseline scenario, we embed selected business and user attributes into Review. This introduces redundancy, but reduces the number of repeated $lookup stages. Such a redesign is common when specific query patterns become performance-critical. From a workload perspective, this scenario illustrates that a shift in access patterns—for instance, an increased emphasis on review-centric queries that aggregate or filter business and user attributes—can motivate a change in the logical representation while leaving the conceptual schema unchanged. The framework generates this adapted representation through mapping modifications alone, without requiring re-inference or schema reconstruction. Figure 13 illustrates the embedding transformation. The schema category remains unchanged; the change is applied at the mapping level. Concretely, we embed business attributes (state, city, name) and user name into the Review mapping. Queries Q2, Q5, and Q6 are modified accordingly. Figure 14 shows that embedding substantially reduces query costs for performance-critical patterns and makes execution times more comparable across representations. This scenario demonstrates that TransforMMer can generate controlled benchmark variants reflecting an evolution step driven 32
Review Business Checkin
Review User
Tip
Business
{;}
Checkin
User Tip
{;}
JSON
attribute
Review
Business Checkin
User
Review
Tip
Business
Checkin
User Tip
{;}
Figure 13: Transformations of the Yelp dataset to embedded MongoDB (and PostgreSQL).
33
Figure 14: Average execution times of the queries from Table 2 on the Yelp dataset, comparing Embedded MongoDB and PostgreSQL. Error bars represent one standard deviation over the last 50 runs per query.
by workload considerations. Scenario C: System Evolution through Data Integration and Enrichment. This scenario reflects the evolution of the system through the integration of additional data sources. We demonstrate multi-model input integration by enriching Yelp business listings with US city information. This reflects a common information systems scenario in which additional data sources are integrated over time. Figure 15 shows the enrichment and the resulting transformations. We embed selected USCities fields into Business (population, density, timezone). Table 3 lists the query set. Figure 16 shows that MongoDB tends to be faster on simple filter-heavy queries such as Q8 and Q10, likely due to embedding that reduces join-like processing at query time. PostgreSQL is more stable for aggregation-heavy queries such as Q9 and Q12, where optimizer and aggregation strategies are more effective. This scenario illustrates how enrichment changes the workload and how performance trade-offs depend on representation decisions.
34
Review USCities
Business
User Tip
Checkin
CSV
{;} JSON
attribute
Review
Business
Review User
Business
User
Tip USCities
Tip
Checkin
USCities
Checkin
{;}
Figure 15: Transformations of the Yelp dataset and US Cities Data to embedded MongoDB (and PostgreSQL).
35
Table 3: Experimental Queries for Yelp dataset with US Cities data
Query Q8 Q9 Q10 Q11 Q12
Query Description
Type
Note
Find businesses located in cities with a population over one million. Find cities with more than 100 businesses and a population over 500,000. List all businesses in cities within the Eastern timezone. Count the number of businesses per timezone. Top 10 most densely populated cities sorted by average business rating.
Filter + Join
Enrichment using population Count + filter with city data Join with timezone filter Timezone aggregation Ranking with enrichment
Aggregation + Join Filter + Join Group + Count Aggregation + Sort + Join
5.3.2. BibleData: Relationship-Centric System Representations BibleData provides a rich network of relationships, which makes it suitable for demonstrating graph transformations. Table 4 lists the query set. Scenario D: Choosing Between Document-Oriented and Graph-Oriented System Architectures. This scenario represents an architectural choice for relationship-centric information systems. To demonstrate a graph transformation, we convert a portion of the dataset to Neo4j and compare it with MongoDB. MongoDB is used for its flexible document schema, which is helpful given the dataset’s missing values. We transform only a subset because one part lacks meaningful relationships; this is indicated by the redhighlighted section in Figure 17. Figure 17 shows that the number of kinds remains the same after both transformations, while Neo4j allows some schema objects to be modeled as relationships (shown as capitalized labels). The label plays a dual role depending on the context. Figures 18 and 19 show the measured query costs. MongoDB performs better on simpler queries (Q13–Q17), while Neo4j is more efficient for relationshipintensive queries (Q18–Q20) involving deeper traversal. This scenario demonstrates that the choice of representation (document vs graph) is inherently workload-dependent: MongoDB is more efficient for simpler lookup and aggregation patterns, while Neo4j excels at relationship-intensive traversal workloads. A shift in the system’s access patterns—for example, a growing demand for multi-hop graph traversal queries—would therefore justify 36
Figure 16: Average execution times of the queries from Table 3 on the Yelp dataset and US Cities data, comparing Embedded MongoDB and PostgreSQL. Error bars represent one standard deviation over the last 50 runs per query. Table 4: Experimental Queries for BibleData
Query
Query Description
Type
Note
Q13
Persons mentioned in references describing events in Jerusalem. For each person, count how many unique labels describe them. Persons connected via a relationship of type “father”. Persons mentioned together in the same reference. References where a person is in an event and described with a label. For each label type, count how many persons it describes. Average age of persons at the time of events described in Genesis. Label trail (Label → Verse → Person) for King David.
Filter + Join
Pattern navigation + filtering Counting and grouping Direct traversal
Q14 Q15 Q16 Q17 Q18 Q19 Q20
37
Aggregation Pattern + Filter Join + Group Multi-relation Join Group + Count Join + Aggregation Path Traversal
Co-occurrence detection Multi-edge matching Aggregation across labels Join-heavy aggregation Deep traversal
Event
Relationship
Reference
Person
Alamo
Book Label Commandment
Verse
CSV
Event
Relationship
Event
RELATIONSHIP
Reference
Person
Reference
Person
Label
Verse
LABEL
{;}
VERSE
Figure 17: Transformations of the BibleData dataset to MongoDB and Neo4j.
38
Figure 18: Average execution times of the queries from Table 4 on the Bible dataset, comparing MongoDB and Neo4j. Error bars represent one standard deviation over the last 50 runs per query.
migrating the relevant portion toward a graph representation, a transition that the framework can generate from the same conceptual schema without modifying it. Scenario E: Polyglot Persistence and Hybrid Information System Architecture. Finally, this scenario emulates a polyglot persistence architecture combining multiple specialized data technologies. We emulate a hybrid evolution step by storing different parts of the dataset in separate systems to exploit their respective strengths. We store one part in MongoDB and another in Neo4j. This allows inclusion of the entire dataset, including the previously excluded part, which can be handled efficiently in MongoDB due to its flatter structure. This partitioned design can also be interpreted as a workload-driven adaptation: the parts of the dataset subject to traversalintensive access patterns are assigned to Neo4j, while those accessed primarily through flat lookups reside in MongoDB. Both portions are derived from the same conceptual schema, enabling systematic evaluation of such hybrid workload assignments. Figure 20 illustrates the partitioned transformation and cross-model queries. The green areas represent data stored in MongoDB and the blue areas rep39
Figure 19: Average execution times of the queries from Table 4 on the Bible dataset, comparing MongoDB and Neo4j. Error bars represent one standard deviation over the last 50 runs per query. Time is shown on a logarithmic scale.
resent data stored in Neo4j. The intentional overlap in Label reflects a redundant but practical design choice to support cross-model access patterns. 5.4. Summary of Findings and Framework Capabilities Across the scenarios, the experiments demonstrate the following: • Comparable benchmark datasets can be generated across heterogeneous target systems from a shared conceptual basis (supporting the benchmark engineering objective). • Representation choices and evolution-like redesign steps (embedding, enrichment, partitioning) lead to measurable differences in observed query costs and reveal model-specific trade-offs. • The framework enables controlled exploration of structural variants while maintaining comparability of the underlying data and analytical intent.
40
Event Alamo
Book
RELATIONSHIP
Reference
Commandment
LABEL
Person VERSE
{;}
db.Label.find({ reference_id: { $in: db.Commandment.find({ reference_id: { $in: db.Reference.find( { usx_code: 'EXO' }, { reference_id: 1 } ).map(r => r.reference_id) } }, { reference_id: 1 } ).map(c => c.reference_id) } }, { person_id: 1, _id: 0 })
WITH mongo_ids_array AS ids MATCH (p:Person)-[r:RELATIONSHIP]-(o:Person) WHERE p.person_id IN ids RETURN p.person_id, type(r) AS relationship_type, o.person_id
Figure 20: Partitioned transformation with queries on the BibleData dataset in MongoDB and Neo4j.
41
Transformational Capabilities. Table 5 summarizes the transformation types demonstrated in the scenarios above. Although presented as individual operations, these transformations can be combined to construct more complex benchmark configurations. Table 5: Transformational Capabilities of the Proposed Framework
Transformation Input Model
Output Key Model tions
Cross-model
Single
Single
Multi-Model Generation
Single
Multi
Aggregation / Single Embedding or Multi
Cross model
multi- Multi
Single or Multi Multi
Opera-
Converts between data models Creates multiple views from a single source model Introduces redundancy to reduce querytime joins Unifies sources; reorganizes data across models
Experiments 5.3.1, 5.3.2 5.3.2 5.3.1, 5.3.1
5.3.1
5.5. Answering the Research Questions The experimental findings allow us to answer the research questions formulated in Section 1. Rather than interpreting the results as isolated performance comparisons, we analyze them as evidence on how architectural and representation-level decisions influence behavior of heterogeneous information systems. RQ1: Can multi-model benchmark datasets be systematically generated from heterogeneous real-world data sources using a unified transformation workflow?. The experiments demonstrate that heterogeneous datasets provided in different formats (e.g., JSON and CSV) can be transformed into multiple structurally distinct representations aligned through the same canonical schema and mapping specifications. Across all scenarios, the framework enabled reproducible generation of relational, document, graph, and hybrid multi-model datasets derived from a shared conceptual representation. 42
These findings confirm that systematic multi-model benchmark construction from real-world data is feasible and reproducible. More importantly from an Information Systems perspective, they show that representation alternatives can be engineered in a controlled and comparable manner, providing a structured basis for analyzing architectural design choices. RQ2: How do schema-level transformations and mapping strategies influence observed performance characteristics across heterogeneous systems?. The results indicate that the representation and mapping strategies significantly influence the observed query behavior. Transformations such as embedding, enrichment, and partitioning altered execution characteristics even when the underlying conceptual data and analytical intent remained identical. In several scenarios, the same workload produced substantially different performance profiles depending on the structural representation. These findings confirm that schema-level transformation strategies are not merely implementation details but key architectural factors shaping system behavior. From an Information Systems viewpoint, representation design and workload alignment emerge as central elements in architectural decision-making. RQ3: Can evolution-aware benchmark generation reveal system-level tradeoffs that remain hidden in static benchmark settings?. The experiments show that evolution-like redesign steps—such as attribute embedding, integration of additional data sources, and hybrid partitioning across systems—expose trade-offs that remain invisible in static single-schema evaluations. By generating controlled structural variants derived from the same conceptual basis, the framework makes explicit how changes in representation interact with workload characteristics and system capabilities. These results demonstrate that evolution-aware benchmark construction enables deeper insight into architectural trade-offs than traditional static benchmarking approaches. It supports systematic exploration of design alternatives under conditions that more closely resemble real-world system evolution. Overall, the findings suggest that treating benchmark construction as a transformation-driven engineering process enables systematic and reproducible evaluation of heterogeneous information systems under evolving structural conditions. Rather than ranking database technologies, the experiments illustrate how architectural and representation decisions shape system behavior across the life cycle of an information system. 43
System-Level Interpretation. From an Information Systems perspective, the framework can be interpreted in multiple complementary ways. First, it serves as a benchmark engineering infrastructure that enables consistent preparation of comparable datasets across heterogeneous platforms. Second, it serves as a data integration and transformation layer that supports cross-model restructuring, enrichment, and partitioning—operations commonly required in modern data engineering and ETL workflows. More broadly, the framework positions benchmark generation as an analytical instrument for evidence-based architectural evolution. By making structural transformations explicit and reproducible, it supports systematic reasoning about trade-offs in heterogeneous data architectures. 5.6. Threats to Validity As with any experimental study of complex information systems, the results presented must be interpreted in light of several limitations related to the experimental design, implementation choices, and scope of evaluation. The goal of this section is not to question the observed effects, but to clarify the conditions under which they hold and to outline factors that may influence their generalizability. Because the study focuses on representation-aware and evolution-oriented analysis rather than absolute system benchmarking, the primary risks concern the comparability of scenarios, the interpretation of measured behavior, and the transferability of findings to other architectural contexts. The following paragraphs discuss these aspects in terms of internal, construct, external, and conclusion validity. Internal validity. To mitigate transient measurement effects, the database containers were restarted before each measurement set, queries were executed repeatedly, and an initial warm-up phase was excluded from the analysis. Nevertheless, Docker scheduling variability and background operating system activity may introduce noise. Since our interpretation focuses on relative trends across structurally distinct variants rather than absolute performance values, the conclusions are less sensitive to minor fluctuations. Construct validity. Queries were manually translated into system-specific languages, which can influence execution behavior through formulation and indexing choices. We mitigate this risk by maintaining consistent analytical intent across representations and by ensuring that all systems operate on 44
datasets generated from the same conceptual basis. The purpose is not to demonstrate optimal tuning for each system, but to observe how changes in representation influence system behavior under comparable conditions. External validity. The experiments involve three database systems and selected real-world datasets. While numerical results may vary across platforms or larger-scale deployments, the observed patterns—such as the impact of embedding, enrichment, and partitioning—reflect general architectural trade-offs encountered in heterogeneous information systems. Therefore, the conclusions primarily concern methodological implications rather than performance claims about specific technologies. Conclusion validity. Execution times were averaged over multiple runs and variability is reported using standard deviation error bars. The interpretation emphasizes consistent behavioral trends across scenarios rather than marginal differences between systems. 5.7. DaRe To support transparency, reuse, and longitudinal evaluation of architectural decisions, we maintain a repository called DaRe (Dataset Repository)5 . DaRe stores original datasets, unified schema representations, transformation configurations, mappings, and generated benchmark variants corresponding to the scenarios presented in this paper. By preserving transformation artifacts and version metadata alongside generated datasets, DaRe enables reproducible reconstruction of alternative structural representations. This supports a systematic comparison of architectural and representation-level design decisions in evolving heterogeneous information systems. In this sense, DaRe complements the proposed framework by providing an infrastructure for repeatable evaluation and iterative system evolution, enabling both researchers and practitioners to analyze and extend benchmark scenarios under controlled and documented conditions.
5
https://dare.mmcatdb.com/
45
Figure 21: DaRe repository supporting reproducible benchmark engineering and evolutionaware evaluation of heterogeneous information systems.
46
6. Discussion and Implications for Information Systems The results of this study extend beyond database performance comparison and have broader implications for the design and evolution of heterogeneous information systems. Contemporary information systems rarely operate within a single data model or a static architectural configuration. Instead, they evolve incrementally in response to workload changes, integration demands, scalability requirements, and organizational constraints. Structural redesign—such as embedding, normalization changes, enrichment with external data sources, or partitioning across multiple storage technologies—is therefore a recurring architectural activity rather than an exceptional event. 6.1. Benchmarking as an Architectural Design Instrument Traditional benchmarking approaches evaluate database systems under fixed schemas and predefined workloads. While useful for technology comparison, such static benchmarks provide limited support for architectural decision-making in evolving environments. The presented framework repositions benchmarking as an explicit architectural design-support instrument. By enabling controlled generation of structurally distinct yet semantically aligned dataset variants, the approach allows system architects to explore alternative representation strategies before committing to long-term architectural decisions. Embedding strategies, hybrid partitioning, or cross-model reorganizations can be evaluated systematically rather than introduced through ad hoc experimentation in production systems. From this perspective, evolution-aware benchmarking becomes part of the information systems design cycle. It supports iterative exploration of structural alternatives, makes performance implications observable, and documents transformation decisions in a reproducible manner. 6.2. Representation-Aware Trade-offs in Heterogeneous Systems The experiments demonstrate that the observed system behavior is not determined solely by database technology. Instead, it emerges from the interaction between representation design, workload characteristics, and architectural context. Join-intensive analytical patterns behave differently under normalized relational schemas, embedded document structures, and graphbased representations. Likewise, hybrid layouts combining multiple systems may exploit complementary strengths.
47
For information systems architects, this implies that the selection of persistence technology cannot be separated from modeling decisions. Architectural trade-offs must consider the representation strategy, anticipated workload evolution, integration patterns, and long-term maintainability simultaneously. The proposed framework makes such trade-offs explicit by treating representation-level alternatives as configurable, reproducible transformations rather than as implicit implementation choices. 6.3. Workload Evolution as a Driver of Structural Adaptation Schema changes are a well-recognized driver of structural evolution in heterogeneous information systems. However, changes in application workloads and access patterns represent an equally significant—and in practice often more frequent—source of structural dynamics. New query patterns, shifting analytical demands, or the integration of new data consumers can require restructuring the logical representation of data even when the underlying conceptual schema remains unchanged. The proposed framework explicitly accommodates this class of adaptation. Because mappings between the schema category and the logical representations are treated as first-class, independently modifiable artifacts, alternative logical representations can be generated by modifying mappings alone, without altering the conceptual schema. This decoupling of conceptual stability from representational flexibility is a key design property of the framework. More broadly, for any significant workload change the framework enables systematic exploration of which logical representation best accommodates the new access patterns, evaluation of the associated performance trade-offs, and reproducible generation of the adapted configuration. This extends the framework’s applicability beyond schema-evolution-driven benchmarking to the systematic analysis of workload-induced structural adaptation throughout the operational life cycle of heterogeneous information systems. 6.4. Implications for the Information Systems Life Cycle A further contribution concerns life cycle integration. In practice, information systems undergo continuous redesign: new data sources are integrated, schemas evolve, and storage strategies are adapted. However, the evaluation of these changes is often informal and difficult to reproduce. By storing transformation artifacts, mappings, and dataset versions alongside metadata, the framework enables traceable, repeatable evaluation of 48
architectural modifications. This capability aligns benchmarking with iterative system development processes and enables evidence-based architectural evolution across the design, integration, optimization, and reconfiguration phases. Rather than treating benchmarking as a one-time post-deployment activity, the presented approach integrates it into the broader life cycle of heterogeneous information systems. In this sense, transformation-driven benchmark engineering provides a structured bridge between conceptual data modeling and empirical system evaluation. 6.5. Positioning Within Information Systems Research Within the Information Systems discipline, this work’s contribution lies in its methodological shift. Instead of proposing a new database engine or optimization technique, the framework introduces a structured process for analyzing architectural alternatives in evolving multi-model environments. It operationalizes representation-aware experimentation and connects formal modeling foundations with practical system-level evaluation. The study therefore contributes to research on system design methodologies, architectural trade-off analysis, life-cycle-aware evaluation, and reproducible experimentation in heterogeneous information systems. By formalizing structural transformations and making them experimentally observable, the framework supports systematic reasoning about architectural decisions under evolving conditions.
49
7. Conclusion This work positions evolution-aware benchmark construction as a designsupport methodology for heterogeneous information systems rather than merely as a performance comparison tool. In modern multi-model environments, architectural decisions regarding data representation, integration, and storage strategies directly influence system behavior, maintainability, and the cost of evolution. However, such decisions are often evaluated informally or retrospectively. We introduced TransforMMer, a transformation-driven framework grounded in a unified structural representation that enables systematic generation of comparable benchmark variants across relational, document, and graph systems. By treating schema transformations and representation changes as explicit, configurable artifacts, the framework supports reproducible exploration of architectural alternatives. The experimental scenarios demonstrated that evolution-like redesign steps—such as embedding, enrichment, and partitioned storage—produce measurable and explainable differences in observed behavior. Crucially, these differences arise from the interaction between workload characteristics and representation design, rather than from database technology alone. This highlights a broader applicability of the framework: beyond propagating conceptual schema changes to logical representations and queries, it also supports workload-driven adaptation of logical representations while preserving the underlying schema—a scenario that is arguably more frequent in production systems than schema restructuring itself. This highlights the importance of representation-aware evaluation in heterogeneous information systems. Beyond its technical mechanisms, this work’s primary contribution is methodological. It reframes benchmarking as an integral component of the information systems life cycle, enabling controlled experimentation with structural alternatives before making architectural commitments. By integrating conceptual modeling, structural transformation, and empirical evaluation into a coherent process, the proposed approach supports evidence-based architectural evolution in complex data ecosystems. Future work will extend the framework with automated query transformation, unified workload specification, systematic benchmarking of workloaddriven structural adaptations, and broader evaluation across distributed deployments. We envision transformation-driven benchmark engineering as a reusable methodological foundation for systematic architectural analysis and 50
life-cycle-aware evolution of heterogeneous information systems.
51
Acknowledgment Supported by the GAČR grant no. 23-07781S. References [1] Transaction Processing Performance Council, Tpc benchmarks overview, https://www.tpc.org/information/benchmarks5.asp, accessed: 2026-02-22 (2026). URL https://www.tpc.org/information/benchmarks5.asp [2] O. Erling, A. Averbuch, J. Larriba-Pey, H. Chafi, A. Gubichev, A. Prat, M.-D. Pham, P. Boncz, The ldbc social network benchmark: Interactive workload, SIGMOD ’15, Association for Computing Machinery, New York, NY, USA, 2015, pp. 619–630. [3] B. F. Cooper, A. Silberstein, E. Tam, R. Ramakrishnan, R. Sears, Benchmarking cloud serving systems with ycsb, in: Proceedings of the 1st ACM Symposium on Cloud Computing, SoCC ’10, Association for Computing Machinery, New York, NY, USA, 2010, pp. 143–154. [4] A. Schmidt, F. Waas, M. Kersten, M. J. Carey, I. Manolescu, R. Busse, XMark: A Benchmark for XML Data Management, in: Proceedings of the 28th International Conference on Very Large Data Bases (VLDB), Morgan Kaufmann, Hong Kong, China, 2002, pp. 974–985. [5] Y. Guo, Z. Pan, J. Heflin, LUBM: A benchmark for OWL knowledge base systems, in: International Semantic Web Conference, Springer, 2005, pp. 123–138. [6] C. Marujo, Synthetic data generation with sdv tutorial, https://www.kaggle.com/code/mcarujo/ synthetic-data-generation-sdv-tutotial, kaggle notebook. Accessed: 2026-02-22 (2023). URL https://www.kaggle.com/code/mcarujo/ synthetic-data-generation-sdv-tutotial [7] EaseCloud, Faker data generator, https://www.easecloud.io/tools/ database/faker-data-generator/, accessed: 2026-02-22 (2026). URL https://www.easecloud.io/tools/database/ faker-data-generator/ 52
[8] G. Bagan, A. Bonifati, R. Ciucanu, G. H. L. Fletcher, A. Lemay, A. PratPérez, gMark: Schema-driven graph query benchmark generation, IEEE Transactions on Knowledge and Data Engineering 29 (4) (2016) 856–869. [9] A. Ghazal, T. Rabl, M. Hu, F. Raab, M. Poess, A. Crolotte, H.-A. Jacobsen, BigBench: Towards an Industry Standard Benchmark for Big Data Analytics, in: Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data, ACM, New York, NY, USA, 2013, pp. 1197–1208. [10] C. Zhang, J. Lu, Holistic evaluation in multi-model databases benchmarking, Distributed and Parallel Databases 39 (1) (2019) 1–33. [11] B. Kim, K. Koo, U. Enkhbat, S. Kim, J. Kim, B. Moon, M2Bench, Proceedings of the VLDB Endowment 16 (4) (2022) 747–759. [12] D. Lengweiler, M. Vogt, H. Schuldt, MMSBench-Net: Scenario-Based Evaluation of Multi-Model Database Systems, in: Proceedings of the 34th GI-Workshop on Foundations of Databases, University of Basel, 2023. [13] P. Koupil, I. Holubová, A unified representation and transformation of multi-model data using category theory, Journal of Big Data 9 (1) (may 2022). [14] I. Holubová, A. Šrůtková, J. Bártík, Reshaping Reality: Creating MultiModel Data and Queries from Real-World Inputs, in: Proceedings of the 20th International Conference on Evaluation of Novel Approaches to Software Engineering, ENASE 2025, SCITEPRESS, Porto, Portugal, 2025, pp. 174–184. [15] J. Bártík, A. Šrůtková, I. Holubová, TransforMMer: A Universal MultiModel Data Generator, in: Proceedings of the 28th International Conference on Extending Database Technology (EDBT), OpenProceedings.org, Barcelona, Spain, 2025, pp. 1150–1153. [16] P. Koupil, S. Hricko, I. Holubová, A universal approach for multi-model schema inference, Journal of Big Data 9 (1) (aug 2022). [17] M. Zaharia, M. Chowdhury, M. J. Franklin, S. Shenker, I. Stoica, Spark: Cluster Computing with Working Sets, in: Proceedings of the 2nd 53
USENIX Conference on Hot Topics in Cloud Computing (HotCloud’10), 2010, p. 10. [18] P. Koupil, J. Bártík, I. Holubová, MM-evocat: A Tool for Modelling and Evolution Management of Multi-Model Data, in: Proceedings of the 31st ACM International Conference on Information; Knowledge Management, CIKM ’22, ACM, New York, NY, USA, 2022. [19] P. Koupil, D. Crha, I. Holubová, A Universal Approach for Simplified Redundancy-Aware Cross-Model Querying, Information Systems 127 (2025) 102456. URL https://doi.org/10.1016/j.is.2024.102456 [20] W. Felter, A. Ferreira, R. Rajamony, J. Rubio, An updated performance comparison of virtual machines and Linux containers, in: 2015 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), IEEE, 2015, pp. 171–172. [21] T. Zhang, Y. Pan, Yelp Challenge Project Report, https://www.yelp. com/dataset_challenge/, project report based on the Yelp dataset challenge (2014). [22] The PostgreSQL Global Development Group, PostgreSQL: The World’s Most Advanced Open Source Relational Database, version 16 (2024). URL https://www.postgresql.org [23] M. Alam, B. Cevallos, O. Flores, R. Lunetto, K. Yayoshi, J. Woo, Yelp Dataset Analysis using Scalable Big Data, supported by Oracle Cloud Innovation Accelerator (2021). URL https://arxiv.org/abs/2104.08396 [24] MongoDB Inc., MongoDB: The Developer Data Platform, version 7.0 (2024). URL https://www.mongodb.com [25] A. Makris, K. Tserpes, G. Spiliopoulos, D. Anagnostopoulos, MongoDB vs PostgreSQL: A comparative study on performance aspects, GeoInformatica 24 (2) (2020) 243–268. URL https://doi.org/10.1007/s10707-020-00407-w
54