DIADA: Automatic Data Composition in Data Lakes

DIADA: Automatic Data Composition in Data Lakes Marc Maynou

Universitat Politècnica de Catalunya Barcelona, Spain marc.maynou@upc.edu

Albert Martin

Universitat Politècnica de Catalunya Barcelona, Spain albert.martin.g@upc.edu

Anna Queralt

arXiv:2610.01646v1 [cs.DB] 1 Oct 2026

Universitat Politècnica de Catalunya Barcelona, Spain anna.queralt@upc.edu

Sergi Nadal

Universitat Politècnica de Catalunya Barcelona, Spain sergi.nadal@upc.edu

Oscar Romero

Universitat Politècnica de Catalunya Barcelona, Spain oscar.romero@upc.edu

ABSTRACT

1

Data lakes contain a plethora of attributes scattered across many tables that, when combined, provide enhanced assets for data analysis. Nonetheless, deciding which attributes belong together in meaningful relations remains a manual, per-task effort. Merging by joinability alone provides no guarantees regarding attribute relevance, while selecting features against a single target discards attributes useful to other tasks. To address this gap, we introduce the data composition problem: organizing a fragmented, heterogeneous lake into meaningful relations, agnostic of any particular analytical task so that the resulting organization can serve as a common foundation for diverse downstream analyses. We propose DIADA, a composition system that employs multivariate dependence as the criterion for assessing the meaningfulness of a relation and approximates it by hypothesizing independence among attributes and identifying those sets that violate this hypothesis. To do so, we map the attributes to a predicate space, forming a lattice under inclusion and mining those predicate sets that exhibit dependence among their constituents. We contribute a dedicated and scalable algorithm to effectively explore this space, outscaling classical algorithms for mining relationships, thus discovering dependencies that would otherwise be impractical to identify. We demonstrate that applying a single data composition process benefits diverse potential downstream tasks. This is the result of providing a subset of low-noise, statistically relevant attributes that increases the confidence that detected patterns are grounded in real relationships, thus preventing common modeling issues in large-scale environments.

Modern data-driven organizations store their data in massive, heterogeneous repositories known as data lakes [21]. Their flexible ingestion policies, however, often turn these repositories into poorly organized data swamps where finding and preparing relevant data becomes cumbersome [38]. To mitigate this problem, organizations structure their data management pipelines into successive curation zones (e.g., the notions of bronze, silver, and gold layers used in the industry [14]) where data assets undergo increasing levels of refinement before being made available for downstream analytics [3]. However, automatically organizing the resulting assets to favor their subsequent use is a challenging task. We focus on this particular stage of the process: once data from different sources have been homogenized, how their attributes should be integrated into relevant relations for downstream analysis. We call this problem data composition: organizing the attributes of a fragmented data lake into meaningful relations. Consider Figure 1a, which depicts a simplified data lake whose attributes are color coded according to their relevance to a specific target variable. For example, both the num_rooms of a property and the tax_rate of its neighborhood are likely to be informative to predict its market_value, whereas agent_phone_num is not related to either market_value or any other attribute. A naive approach to structure the data would run a data discovery task to identify candidate join attributes and perform a multiway join (Figure 1b) that combines potentially useful information scattered across datasets into a single relation. However, data discovery may propose spurious joins, such as property_id and station_id, alongside meaningful ones, such as the join on postal_code. Consequently, the relation supplied to an ML pipeline predicting, say, market_value, would contain inert attributes (gray) and an entire group of weather attributes (blue) irrelevant to this particular prediction task. Ideally, the joined table should be reorganized as shown in Figure 1c, with intrinsically related attributes composed into separate relations. The task of data composition is mainly performed in a manual, slow and unreliable fashion by domain experts through the execution of handcrafted joins and partitions [1]. Alternatively, there are many research fields that propose automated methodologies to organize the attributes and relations of a data repository, each presenting a different way of characterizing the meaningfulness of the final relations. Nonetheless, all have important limitations towards performing data composition tasks. Functional dependency (FD) discovery identifies attribute sets that determine others [25],

PVLDB Reference Format: Marc Maynou, Albert Martin, Sergi Nadal, Anna Queralt, and Oscar Romero. DIADA: Automatic Data Composition in Data Lakes. PVLDB, 20(1): XXX-XXX, 2027. doi:XX.XX/XXX.XX PVLDB Artifact Availability: The source code, data, and/or other artifacts have been made available at https://github.com/dtak-upc/DIADA. This work is licensed under the Creative Commons BY-NC-ND 4.0 International License. Visit https://creativecommons.org/licenses/by-nc-nd/4.0/ to view a copy of this license. For any use beyond those covered by this license, obtain permission by emailing info@vldb.org. Copyright is held by the owner/author(s). Publication rights licensed to the VLDB Endowment. Proceedings of the VLDB Endowment, Vol. 20, No. 1 ISSN 2150-8097. doi:XX.XX/XXX.XX

INTRODUCTION

Figure 1: The data composition problem illustrated. A data lake (a) contains tables that can be used to satisfy predictive tasks. A multiway join is executed for these tables (b) to place all relevant information in a single asset, but a spurious join introduces unhelpful features in the dataset. Ideally, joined data should be separated into specific relations (c) to benefit any potential downstream task. In practice, this process is either done manually (costly and prone to errors) or substituted by automated techniques (d), which either select features for a specific target (and discard the rest) or preserve noisy data. which can guide the separation of datasets into normalized relations. However, these constraints are deterministic, and the classical approaches to detect them are syntactic [40] and scale poorly [26], making them not suitable for data lakes’ heterogeneity and large size. Data discovery [19, 38] presents semantic and scalable techniques to identify tables that are related and can be combined, but does not assess the relevance of the attributes in the relation. This leads to noise not being filtered out, as the spurious join in Figure 1b illustrates. To address this, we can introduce techniques to select the relevant subset of attributes out of the combined data. Figure 1d exemplifies two main families of such approaches. On one hand, the selection of attributes can be guided by the association to a predictive target (i.e., target-driven, such as data augmentation and supervised selection [10, 11, 27]). However, assessing attribute relevance against a given target discards whatever is not associated with it, yet these deleted attributes might provide information that a different task may need (e.g., blue features for predicting monthly_temp). Thus, the costly selection process must be executed anew for every new target. On the other hand, target-agnostic approaches (e.g., unsupervised feature selection [36, 46]) circumvent this problem by not conditioning attribute importance on a target, instead seeking to preserve some structural property of the data (e.g., cluster structure). However, as they do not explicitly assess the degree of shared information across attributes, uninformative attributes may be preserved if they contribute to the original and potentially noisy structure being captured. At the core of this problem lies the need for assessing the relatedness of sets of attributes and, therefore, guide the separation of features into strongly intra-related groups. We argue that relatedness is, fundamentally, a question of shared information: two

attributes are related when knowledge of one changes what is expected of the other. This notion extends naturally from pairs of attributes to arbitrary sets through their multivariate dependence [49], that is, the extent to which each attribute is conditioned by all the others jointly. This dependence is precisely what analytical tasks look for, as it allows patterns to be defined and exploited to extract insight or perform inference. A meaningful relation is, therefore, one whose attributes exhibit strong multivariate dependence. However, metrics that fully compute such a notion (i.e., total correlation [49] and its variants [35]) are exponential to estimate due to the combinatorial explosion of assessing increasingly large sets of features jointly [5]. Thus, current methods to compute multivariate dependence either fail to scale or opt exclusively for pairwise dependencies. Take a property’s floor_number, its district’s flood_risk, and its market_value from Figure 1: the marginal effects of both floor_number and flood_risk over market_value are likely to be minimal and, hence, missed. Nonetheless, ground floors are cheaper where flooding is a risk. Thus, only when both are observed does the strong dependence with market_value appear. In this work we aim at providing an approach that efficiently approximates multivariate dependencies across sets of attributes at scale. This methodology is encapsulated in DIADA, the first system that performs automatic data composition based on evaluating the amount of shared information across sets of attributes. To do so, DIADA hypothesizes that all attributes in the relation are independent, and its goal is to detect which groups of attributes violate that independence hypothesis. DIADA uses an intermediate abstraction layer that decouples raw data formats from multivariate dependency analysis, allowing dependencies to be discovered across heterogeneous data. More precisely, every attribute is mapped to a set of predicates via a Boolean function over a chosen domain 2

Table 1: Requirements on the composition criterion. ✓: satisfied, ◦: partially, ✗: not addressed.

(e.g., individual tuples or pairs of tuples). The combination of these predicates results in a space that forms a lattice under inclusion. DIADA then mines this lattice to extract predicate sets whose joint probability departs from what is expected by assuming predicate independence. Such a departure indicates that the corresponding attributes are related and should be composed into the same relation. DIADA, thus, approximates multivariate dependence by applying a statistical test regarding the independence of predicates. Exploring such a lattice reduces to to the well-studied problem of mining frequent itemsets. Nonetheless, scaling this search to an entire data lake still introduces computational challenges both in the number of rows (i.e., horizontal, as there are more predicates per attribute) and columns (i.e., vertical, as there are more combinations of predicates). To address this, DIADA introduces an approximation factor, a statistically validated ceiling on how uncertain any estimated probability may remain. Thus, DIADA performs an informed and traceable search that prunes the space by navigating only towards predicate sets whose relatedness is still highly uncertain. Consequently, our approach provides a statistics-based definition of meaningful relations in this context, and contributes an efficient and scalable method to approximate multivariate dependencies that allow us, for the first time, to operationalize the problem of data composition. Our experiments showcase how these optimizations allow DIADA to outscale traditional miners in the context of data lakes and discover dependencies that could otherwise be impractical to detect. We prove that in realistic scenarios DIADA correctly filters noisy attributes from relations while preserving relevant attributes, which other approaches are not able to do. We also demonstrate that not composing data prior to executing downstream tasks negatively impacts the final models, as scalable analytical pipelines are not able to effectively preserve relevant data. By executing DIADA a single time, diverse downstream models improve both their performance and generalization, regardless of their target.

R2

R3

R4

Join / union discovery Lake organization Augmentation / supervised FS Unsupervised feature selection FD / key discovery Statistical dependency discovery Schema decomposition

✓ ✓ ✗ ✓ ✓ ✓ ✓

✗ ✗ ◦ ◦ ✗ ✗ ✓

✗ ◦ ✗ ◦ ✓ ✓ ✓

✓ ◦ ◦ ✓ ✗ ◦ ✗

DIADA

✓

✓

✓

✓

variable. (R2) Multivariate: composition must be guided by the relatedness of sets of attributes, not merely attribute pairs. (R3) Complete: every attribute must be placed in some relation. (R4) Lake-scale: executing the composition must remain feasible as the lake grows. In the remainder of this section, we review the main families of approaches that are relevant to data composition. Table 1 summarizes the extent to which each family satisfies the stated requirements.

2.1

Assembling data from a lake

Data discovery. Join and union discovery locate tables that can be combined with a given base table through value overlap, containment or embedding similarity [7, 19, 34, 38, 39]. These methods answer whether two tables can be combined, not whether the attributes they contribute form a meaningful relation (R2, R3). Their goal is to identify related assets, not to evaluate the relevance of the attributes. Discovery is therefore the natural upstream stage for DIADA, and we use it as such (Section 5). Lake organization. A smaller line organizes the lake itself rather than serving individual queries. Aurum builds an enterprise knowledge graph linking columns by syntactic and semantic similarity [18], and navigation-oriented approaches induce a hierarchy over tables so that users can browse a lake without knowing its contents [37]. These share our objective of a single reusable organization, but they organize tables by high-level similarity, rather than through the relatedness of their attributes (R2).

Contributions. Our contributions are as follows: • We define the data composition problem: how to organize a fragmented lake into meaningful relations, agnostic of any target attribute. We take multivariate dependence as the composition criterion. • We introduce an approximate and scalable algorithm that mines a predicate-set lattice to detect sets of dependent attributes, outperforming classical methods for frequent itemset mining as attributes grow, discovering dependencies that would be otherwise impractical. • We present DIADA: the first system for automatic data composition, which implements the above algorithm. We showcase that applying a single, target-agnostic composition task improves the utility and robustness of downstream modeling pipelines regardless of the target attribute, as composing generates meaningful datasets not conditioned on a given process.

2

R1

Data augmentation and supervised selection. Supervised feature selection aims to identify a subset of attributes that maximizes predictive performance for a defined target variable, whereas data augmentation expands the search space to an entire data lake by integrating additional data sources before feature selection [10, 11, 13, 20, 27]. Because relevance is defined with respect to that target, the search is repeated whenever the analytical goal changes, and attributes useful to later tasks are discarded rather than organized (R1, R3).

2.2

Unsupervised feature selection

Unsupervised feature selection (UFS) reduces dataset dimensionality while preserving a surrogate objective over the data [36, 46]. Methods differ in the structure they maintain: local geometry [23, 29, 54], cluster assignment [9, 51], matrix reconstruction [4, 55], and global variance [30, 52]. Their main limitation is that their structure-preserving objectives assume the structure is sound to

RELATED WORK

The task of data composition can be further specified through four requirements, which need to be simultaneously fulfilled. (R1) Targetagnostic: composition cannot be defined with respect to a target 3

3

begin with. Hence, if the input contains correlated noise, the objective preserves exactly what should be removed. Nonetheless, it is the only family of methods that fulfills, to some extent, all four composition requirements. Thus, in Section 6.2, we quantify their suitability for the composition problem.

2.3

Discovering dependencies

Exact dependencies. Functional dependency (FD) discovery identifies attribute sets that determine others, and underpin normalization, cleaning and query optimization [25, 40]. Modern FD discovery algorithms scale to tables of realistic size, but still have combinatorial worst-case behavior (R4). Moreover, FDs express deterministic constraints: an exact FD is invalidated by a single counterexample tuple, while the absence of an FD does not imply statistical independence. Consequently, FD and key discovery do not capture the more general multivariate relatedness between attributes (R2).

3.1

Preliminaries

Let 𝑅(𝐴1, 𝐴2, . . . , 𝐴𝑚 ) be a relational schema with attributes 𝐴1, 𝐴2, . . . , 𝐴𝑚 , each taking values over domains 𝐷 1, 𝐷 2, . . . , 𝐷𝑚 , respectively. A relation instance over this schema is a finite set 𝑟 (𝑅) ⊆ 𝐷 1 × 𝐷 2 × · · · × 𝐷𝑚 , where each tuple 𝑡 ∈ 𝑟 (𝑅) is a function mapping every attribute to a value in its corresponding domain, i.e., 𝑡 (𝐴𝑖 ) ∈ 𝐷𝑖 . A core assumption in statistics and machine learning is that the tuples 𝑡 = (𝑎 1, 𝑎 2, . . . , 𝑎𝑚 ) of a relation are drawn from an unknown multivariate distribution over 𝐷 1 × 𝐷 2 × · · · × 𝐷𝑚 [6]. When the attributes 𝐴1, 𝐴2, . . . , 𝐴𝑚 are independent, the joint probability of drawing any given tuple is reduced to the product of marginal attribute probabilities: Ö 𝑝 (𝐴1 = 𝑎 1, 𝐴2 = 𝑎 2, . . . , 𝐴𝑚 = 𝑎𝑚 ) = 𝑝 (𝐴𝑖 = 𝑎𝑖 ).

Statistical dependency discovery. The rigidity of exact constraints can be relaxed to discover broader statistical relationships between attributes. CORDS mines correlations and soft functional dependencies from samples [26], and Zhang et al. give a statistical account of FD discovery under noise [53]. Both are target-agnostic and cheap enough for large tables, but they still lack the simultaneous assessment of the joint relatedness of sets of features (R2).

𝑖

The goal of multivariate analysis is to discover sets of attributes that deviate from this independence. As argued in Section 1, these deviations are what makes a relation worth preserving in data composition, as the shared information between attributes defines patterns that can be exploited by downstream tasks. Computing exact measures of multivariate dependence is unfeasible in the context of data lakes, so the criterion we require cannot be defined through them. It must instead make decisions under sampling while properly accounting for the associated uncertainty, be data-agnostic so as to operate on heterogeneous data, and provide objective measures of association that do not rely on arbitrary thresholds. The remainder of this section constructs this criterion.

Schema decomposition. Maimon [28] mines approximate acyclic schemes using information-theoretic measures, and is the only approach meeting the first three requirements. It solves, however, the dual problem. It refines a relation that already exists into a scheme that preserves it, so the set of attributes that belong together is an input and the question is how to split it. Additionally, Maimon is limited in terms of scalability (R4), its cost being exponential in the number of attributes and reported empirically up to 30 columns.

2.4

A STATISTICAL CRITERION FOR MEANING

This section develops the statistical criterion DIADA employs to quantify the meaningfulness of a set of attributes. First, Section 3.1 fixes notation and the independence baseline. Then, Section 3.2 introduces the predicate space that decouples heterogeneous data from multivariate dependency analysis, Section 3.3 identifies meaningfulness as a statistical hypothesis test with independence as its null, and Section 3.4 formalizes the data composition problem.

Defining the composition criterion

To effectively perform composition tasks it is necessary to establish and evaluate a statistical criterion for attribute relatedness. To that end, we position DIADA against two bodies of work.

3.2

Multivariate measures. Total correlation [49], interaction information, and its variants [35] quantify the divergence between a joint distribution and the product of its marginals, and are the measures composition would ideally adopt. Estimating them is exponential in the number of variables [5], which is precisely what places any criterion defined through them outside (R4), and why scalable methods fall back on pairwise or greedy conditional formulations [8, 48]. Section 3.1 examines these measures in the detail our framework requires and derives the alternative we adopt.

Predicate spaces

Our framework generalizes statistical hypothesis testing to large, heterogeneous data lakes through an intermediate abstraction layer that decouples raw data from dependency analysis. It decomposes the problem of multivariate dependency discovery in two: • Predicate space construction: Given a relational schema of the form 𝑅(𝐴1, 𝐴2, . . . , 𝐴𝑚 ), generate an abstraction layer known as the predicate space. • Predicate association mining: Given the predicate space and a relation instance 𝑟 (𝑅), identify the multivariate dependencies present in the data.

Pattern mining. Once dependence is expressed over predicates, the search space is the itemset lattice, for which Apriori [2] and FP-Growth [22] are the canonical exploration algorithms. Samplingbased variants bound the deviation of the estimated supports [43, 47]. We reuse the lattice but not the pruning principle, since we leverage independence rather than frequency. Certain approaches carry error control through significance testing with family-wise error correction [50] and permutation-based methods [41].

This decomposition yields two complementary advantages. From a data management perspective, data ingestion does not need to adapt the data format as required by downstream dependency analysis, which is of particular value in heterogeneous data lakes. From an analytical perspective, all dependency analyses operate over a common abstraction layer, enabling a broad range of established techniques to be applied independently of the underlying domain 4

Table 2: Predicate spaces for the predicate domain of tuples Ω = 𝑟 (𝑅) and tuple pairs Ω = 𝑟 (𝑅) × 𝑟 (𝑅).

Ω =𝑟 Ω =𝑟 ×𝑟

Name String

Salary Real

= ”𝐴𝑙𝑖𝑐𝑒” ℎ𝑎𝑠 ℎ𝑦𝑝ℎ𝑒𝑛 = 𝑁 𝑎𝑚𝑒 ≠ 𝑁 𝑎𝑚𝑒

∈ [0, 8000] > 8000 > 𝑆𝑎𝑙𝑎𝑟 𝑦 < 𝑆𝑎𝑙𝑎𝑟 𝑦

𝑝 (𝑃𝑖 | 𝑃 ′ \ {𝑃𝑖 }) = 𝑝 (𝑃𝑖 | 𝑄 \ {𝑃𝑖 })

for all 𝑃𝑖 𝑖 ∈ 𝑄. The goal of this subproblem is to enumerate sets of predicates that deviate from this independence, thus rejecting the null hypothesis and implying the existence of a relationship between their respective attributes. This is achieved by statistically modeling conditional distribution terms like 𝑝 (𝑃𝑖 | 𝑃 𝑗 ) and 𝑝 (𝑃𝑖 ), and evaluating the likelihood of their equality under the null hypothesis. Two propositions ground this reasoning.

Message Text

𝑖𝑠 𝑝𝑒𝑟𝑠𝑜𝑛? 𝑖𝑛 𝑐𝑜𝑙𝑜𝑟 ? ∼ 𝐻𝑢𝑒 𝑠𝑝𝑒𝑐𝑡𝑟𝑢𝑚 ∼ 𝐸𝑚𝑏𝑒𝑑𝑑𝑖𝑛𝑔

𝑠𝑖𝑧𝑒 < 256 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒 ∼ 𝑉 𝑜𝑐𝑎𝑏𝑢𝑙𝑎𝑟 𝑦 ∼ 𝐸𝑚𝑏𝑒𝑑𝑑𝑖𝑛𝑔

diversity. We address the first subproblem here, and turn a relation’s meaningfulness into a statistical question in Section 3.3. We define a predicate 𝑃𝑖 : Ω → {0, 1} as any function mapping elements of a space Ω, called the predicate domain of 𝑃𝑖 , to a binary outcome. For each 𝐴𝑖 ∈ 𝑅, we denote by 𝑃 (𝐴𝑖 ) = {𝑃𝑖 , 𝑃 𝑗 , . . . } a set of predicates defined entirely in terms of that attribute. The predicate Ð space is then the union of all such sets, 𝑃 = 𝐴𝑖 ∈𝑅 𝑃 (𝐴𝑖 ), and its construction is the goal of this subproblem. The first row of Table 2 illustrates a predicate space whose domain is individual tuples, and demonstrates the flexibility of this framework in accommodating heterogeneous data. Categorical and binned numerical attributes are handled naturally. Beyond these, the framework allows the definition of concise, semantically meaningful predicates to support complex types such as images or text, which can be incorporated into the multivariate analysis without special treatment. The principal limitation of projecting data into binary predicates is the inevitable loss of information, making the approach unsuitable for tasks requiring a detailed characterization of dependencies. However, since the framework aims to determine whether a dependency exists rather than quantify its nature, this trade-off is acceptable. Moreover, the abstraction extends beyond individual tuples. The second row of Table 2 shows how it generalizes to pairs of tuples, yielding new spaces that have proven useful in other fields [12, 42]. Our implementation instantiates the pair-of-tuples domain, whereas the non-tabular predicates in Table 2 illustrate the broader reach of the abstraction and are left to future work.

3.3

Proposition 1 (Soundness of the abstraction). Let 𝑃1, . . . , 𝑃𝑙 ∈ 𝑃 be predicates over distinct attributes 𝐴𝑥 1 , . . . , 𝐴𝑥𝑙 . If 𝑝 (𝑃1 ∧ · · · ∧ Î 𝑃𝑙 ) ≠ 𝑖 𝑝 (𝑃𝑖 ), then 𝐴𝑥 1 , . . . , 𝐴𝑥𝑙 are not mutually independent. The converse does not hold. Proof. Since each 𝑃𝑖 is derived strictly from its corresponding attribute 𝐴𝑥𝑖 , independence among the attributes guarantees independence among the predicates. Therefore, if the predicates’ joint probability fails to factorize (i.e., it does not equal the product of their individual probabilities), the underlying attributes must be dependent. The converse does not hold, because predicates are lossy abstractions and, thus, if an attribute dependency is entirely contained within the values mapped to a single predicate outcome (e.g., variations inside a single numerical bin), the binary predicates will not detect it, leaving their probabilities perfectly independent. □ Proposition 2 (Equivalence with total correlation). Let the set 𝑃 ′ = {𝑃1, . . . , 𝑃𝑙 } be predicates over pairwise-distinct attributes, and Í let 𝑇𝐶 (𝑃 ′ ) = 𝑖 𝐻 (𝑃𝑖 ) − 𝐻 (𝑃1, . . . , 𝑃𝑙 ) denote the total correlation of the induced Bernoulli vector. Then Equation (2) holds for every subset of 𝑃 ′ iff 𝑇𝐶 (𝑃 ′ ) = 0. Proof. Total correlation (𝑇𝐶) measures the amount of shared information among a set of variables. If 𝑇𝐶 (𝑃 ′ ) = 0, the predicates are perfectly independent, so knowing the probability of some predicates provides no information about the others. That is, every conditional probability equals its marginal probability, which directly satisfies Equation (2). Conversely, suppose Equation (2) holds, meaning that conditioning a predicate on other sets never changes its probability. Expanding the joint probability step-by-step via the chain rule, 𝑝 (𝑃1, . . . , 𝑃𝑙 ) = 𝑝 (𝑃 1 ) · 𝑝 (𝑃 2 | 𝑃1 ) · · · 𝑝 (𝑃𝑙 | 𝑃1, . . . , 𝑃𝑙 −1 ), we can strip the conditions from every term (e.g., 𝑝 (𝑃 2 | 𝑃 1 ) becomes 𝑝 (𝑃 2 )), since Equation (2) guarantees context does not matter. The chain thus collapses into a simple product of individual probabilities, so the joint distribution factorizes perfectly and the variables are entirely independent, so 𝑇𝐶 (𝑃 ′ ) = 0. □

Meaningfulness as a hypothesis test

To determine multivariate dependencies, the framework employs a statistical test assuming complete attribute independence as the null hypothesis: Ö 𝐻 0 : 𝑝 (𝐴1 = 𝑎 1, 𝐴2 = 𝑎 2, . . . , 𝐴𝑚 = 𝑎𝑚 ) = 𝑝 (𝐴𝑖 = 𝑎𝑖 ), 𝑖

𝐻1 :

𝑝 (𝐴1 = 𝑎 1, 𝐴2 = 𝑎 2, . . . , 𝐴𝑚 = 𝑎𝑚 ) ≠

(2)

∈ 𝑃 ′ and all 𝑄 ⊂ 𝑃 ′ with 𝑃

Attribute and Type Tax Photo Real Image = 0% > 0% > 𝑇 𝑎𝑥 < 𝑇 𝑎𝑥

conditioning on fewer predicates:

Ö

𝑝 (𝐴𝑖 = 𝑎𝑖 ).

𝑖

This statistical foundation allows dependency decisions to be made by evaluating predicates on domain samples, and ensures uncertainty is properly accounted for in the modeled distributions. All conclusions about attribute dependencies are grounded in a principled statistical test, quantifying how unlikely the observed predicate behavior would be under true attribute independence. Crucially, this reasoning operates entirely within the predicate space, remaining agnostic to the heterogeneous data types of the source attributes. Remaining at this level of abstraction does not pose a problem, as, by Proposition 1, any dependency detected in

Let 𝑝 (𝑃𝑖 ) denote the probability that predicate 𝑃𝑖 is satisfied when evaluated on a uniformly drawn element from its predicate domain. Under the null hypothesis, the joint probability of any set of predicates drawn from distinct attributes satisfies the factorization property: 𝑝 (𝑃𝑖 ∧ 𝑃 𝑗 ∧ · · · ) = 𝑝 (𝑃𝑖 ) · 𝑝 (𝑃 𝑗 ) · · ·

(1)

for all 𝑃𝑖 ∈ 𝑃 (𝐴𝑥 ), 𝑃 𝑗 ∈ 𝑃 (𝐴𝑦 ), . . . with 𝐴𝑥 , 𝐴𝑦 , . . . pairwise distinct. This implies that for any set of predicates 𝑃 ′ ⊆ 𝑃, we have conditional independence, where probabilities are not affected by 5

the predicate space guarantees a dependency among the source attributes, regardless of their types.

3.4

an efficient procedure to evaluate each 𝑃𝑖 over all elements of Ω, empirically estimate each conditional probability 𝑝 (𝑃𝑖 | 𝑃 𝑗 , . . . ), and accept those predicate sets whose conditional probabilities differ sufficiently from the independence baseline. However, such approaches face two principal scalability challenges:

Problem statement

To formalize the data composition problem, we first define a pairwise relationship between attributes as follows.

• Row scalability: Regardless of the algorithm design, every element of the domain must be visited at least once. This is a significant constraint for large datasets, particularly given the increased domain size of more complex predicate domains. • Column scalability: The predicate space grows linearly with the number of attributes. Thus, the number of candidate predicate sets is exponential in the number of attributes, posing severe limitations for dependency discovery in large attribute sets.

Definition 1 (Related attributes). Attributes 𝐴𝑖 and 𝐴 𝑗 are related, denoted 𝐴𝑖 ∼ 𝐴 𝑗 , iff there exists a predicate set 𝑃 ′ ⊆ 𝑃, over pairwise-distinct attributes, that violates conditional independence (Equation (2)), with 𝑃 (𝐴𝑖 ) ∩ 𝑃 ′ ≠ ∅ and 𝑃 (𝐴 𝑗 ) ∩ 𝑃 ′ ≠ ∅. In other words, a predicate set showcasing conditional dependence establishes a relationship between every pair of attributes represented within it. In a lake, predicates are evaluated over the instance obtained by joining the input relations along the candidate keys. Data composition is then stated as follows.

Both challenges have well-established solutions in the association rule mining literature that transfer naturally to this setting. Row scalability can be improved through sampling and estimating conditional probabilities from a subset Ω ′ ⊆ Ω [44, 47]. Column scalability can be addressed by imposing minimum support or maximum predicate-set size thresholds to prune the search space [2, 22]. However, these approaches introduce arbitrary hyperparameters (e.g., sampling delimiters or support thresholds), which are undesirable due to the lack of a generalizable and mathematically validated criterion to instantiate them. In this section, we present the core algorithm of DIADA, which adapts these methods to discover independence-breaking predicate sets with minimal computation and without arbitrary hyperparameters (Section 4.1).

Problem 1 (Data Composition). Given relation instances 𝑟 (𝑅1 ), . . . , 𝑟 (𝑅𝑘 ), a set of join candidates 𝐽 , and a predicate space 𝑃 over their attributes, compute the composition C = {𝐶 1, . . . , 𝐶𝑞 } of the nonkey attributes induced by the transitive closure of ∼, materializing each block as a relation and replicating every join key of 𝐽 needed to associate those attributes. That is, the output is a graph where nodes indicate attributes and edges identify meaningful dependencies between attributes, thus generating partitions (or components) of semantically related attributes (we further extend this in Section 5). Unrelated attributes form their own relation rather than being discarded, making the composition complete (R3), and the replicated keys keep the resulting relations mutually joinable. Section 4 develops the algorithm that decides ∼ statistically.

4

4.1

Posterior modeling

Instead of relying on point estimates of probabilities computed from a single, arbitrarily sized subset Ω ′ ⊆ Ω, we model the full distribution of each probability. These distributions are inferred from dynamically sized samples Ω(𝑃𝑖 , 𝑃 𝑗 , . . . ) ⊆ Ω, chosen independently for each predicate set. Treating the evaluation of a predicate on a randomly sampled element of Ω as a Bernoulli random variable, the corresponding population probability is modeled using Bayesian Learning with a Beta posterior,

SCALABLE ALGORITHM

Thanks to the predicate abstraction, the problem of discovering multivariate attribute associations can be reduced to well-studied problems such as frequent itemset mining and association rule mining. Algorithms such as Apriori [2] or FP-Growth [22] may be straightforwardly adapted to discover predicate sets that deviate from independence, and by extension to identify the existence of multivariate attribute dependencies. However, while this flexibility is one of the principal advantages of the predicate abstraction, a dedicated algorithm designed specifically for this problem can achieve significantly better performance. Let 𝑃 = {𝑃 1, 𝑃2, . . . , 𝑃𝑘 } be the predicate space for a predicate domain Ω, where 𝑃𝑖 : Ω → {0, 1} ∀𝑃𝑖 ∈ 𝑃. The space of predicate sets can be represented as a partially ordered set under the subset relation ⊂, as in Figure 2. In this representation, each node corresponds to a predicate set {𝑃𝑖 , 𝑃 𝑗 , . . . } and is associated with its joint predicate probability 𝑝 (𝑃𝑖 , 𝑃 𝑗 , . . . ). Each edge corresponds to the addition of a predicate, {𝑃 𝑗 , . . . } → {𝑃𝑖 , 𝑃 𝑗 , . . . }, and is associated with the conditional probability 𝑝 (𝑃𝑖 | 𝑃 𝑗 , . . . ). Under this representation, identifying conditional probabilities that violate Equation 2 can be translated to finding nodes 𝑁 ⊂ 𝑃 where the conditional probability of the edge adding 𝑃𝑖 , 𝑝 (𝑃𝑖 |𝑁 ), is different from the one from the edge adding 𝑃𝑖 to any subset 𝑁 ′ ⊂ 𝑁 , 𝑝 (𝑃𝑖 |𝑁 ′ ). To discover all such predicate sets, one could adopt search strategies analogous to those of classical association rule mining: design

𝑝 (𝑃𝑖 , 𝑃 𝑗 , . . . ) ∼ Beta(𝛼 + 1, 𝛽 + 1), where  𝛼 = 𝑡 ∈ Ω(𝑃𝑖 , 𝑃 𝑗 , . . . ) 𝑃𝑖 (𝑡) ∧ 𝑃 𝑗 (𝑡) ∧ · · ·

,

𝛽 = |Ω(𝑃𝑖 , 𝑃 𝑗 , . . . )| − 𝛼 . Throughout, we write attr(𝑃𝑖 ) for the attribute over which 𝑃𝑖 is defined, and attrs(𝑃 ′ ) = {attr(𝑃𝑖 ) : 𝑃𝑖 ∈ 𝑃 ′ }. Violations of Equation (2) are determined statistically. Given two Beta posteriors with means 𝜇1, 𝜇2 and variances 𝜎12, 𝜎22 for the two conditional probabilities compared by Equation (2), we define the soundness of the comparison as |𝜇 −𝜇 | 𝑠 = √︃ 1 2 2 2 , the number of pooled standard deviations separating 𝜎1 +𝜎2

the estimates1 [32, 33]. The precision of every estimate is driven by a user-defined threshold 𝜖, the approximation factor. Each posterior is refined (Section 4.2) until its variance falls below 𝜖. A comparison is accepted as a violation when 𝑠 ≥ 𝑠 0 , set to 𝑠 0 = Φ−1 (1 − 0.05) 1 For numerical stability, instead of directly modeling the beta distributions, we model

the logarithm of their odds ratio.

6

{𝑃𝐹 , 𝑃𝑅 , 𝑃𝑉 , 𝑃𝐿 }

{𝑃 𝐹 , 𝑃𝑅 , 𝑃𝑉 } .030

{𝑃𝐹 , 𝑃𝑉 , 𝑃𝐿 }

{𝑃 𝐹 , 𝑃𝑅 , 𝑃𝐿 }

Ω(𝑃 ′ ), the goal is to enlarge each sample, Ω(𝑃 ′ ) ← Ω(𝑃 ′ )∪ΔΩ(𝑃 ′ ), and update the corresponding posterior parameters without explicitly storing Ω(𝑃 ′ ), which is prohibitively memory-intensive. The algorithm maintains the invariant that all variances remain approximately equal. Equivalently, distributions are refined at a rate proportional to their current uncertainty, preventing computation from being wasted on already well-saturated probabilities while focusing effort on poorly estimated ones. At each iteration, the algorithm identifies all explored predicate sets whose posterior variance still exceeds the approximation factor 𝜖. If no such predicate sets remain, refinement is complete for the current explored set. The algorithm terminates once lattice expansion (Section 4.3) also produces no new candidates (Section 4.4). A new domain sample ΔΩ ⊆ Ω is then generated.3 This sample is propagated through the explored lattice by filtering it along each edge, retaining only those elements satisfying the predicates encountered along the path4 . Consequently, when visiting a predicate set 𝑃 ′ , the algorithm updates 𝛼 (𝑃 ′ ) and 𝛽 (𝑃 ′ ) using the number of sampled elements that satisfy, or fail to satisfy, all predicates in 𝑃 ′ .

locker_number is unrelated, so its paths are pruned

{𝑃𝑅 , 𝑃𝑉 , 𝑃𝐿 }

𝑝 (𝑃𝑉 |𝑃 𝐹 ∧ 𝑃𝑅 ) = .600 {𝑃 𝐹 , 𝑃𝑅 } .050

{𝑃 𝐹 , 𝑃𝑉 } .045

𝑝 (𝑃𝑉 |𝑃 𝐹 ) = .225 {𝑃𝐹 } .200 𝑃 𝐹 → floor_number = 0 𝑃𝑅 → flood_risk > 0.3

{𝑃𝑅 , 𝑃𝑉 } .058

{𝑃 𝐹 , 𝑃𝐿 } .050

{𝑃𝑉 , 𝑃𝐿 } .051

{𝑃𝑅 , 𝑃𝐿 } .063

𝑝 (𝑃𝑉 |𝑃𝑅 ) = .232 {𝑃𝑉 } .202

{𝑃𝑅 } .250

𝑝 (𝑃𝑉 ) = .202 ∅

{𝑃𝐿 } .250 𝑃𝑉 → market_value < 250.000 𝑃𝐿 → locker_number < 50

Figure 2: Snippet of the lattice for the example of Figure 1. Nodes show the posterior mean of the joint probability at termination and edge labels give the conditional probability of the added predicate. The weak conditional signals 𝑝 (𝑃𝑉 | 𝑃𝐹 ) = .225 and 𝑝 (𝑃𝑉 | 𝑃𝑅 ) = .232 deviate from 𝑝 (𝑃𝑉 ) = .202 just beyond the effect-size floor at 𝜖 = 10−5 (𝛿 min ≈ .007), relating floor_number and flood_risk to market_value. Expansion then reaches {𝑃𝐹 , 𝑃𝑅 , 𝑃𝑉 }, where the conditional 𝑝 (𝑃𝑉 | 𝑃𝐹 ∧ 𝑃𝑅 ) = .600 confirms the strong joint dependence (𝑠 ≫ 𝑠 0 ). locker_number relates to nothing, so its supersets are pruned.

4.3

throughout2 . Violating comparisons are the backbone of the algorithm, serving as a guide for exploration (Section 4.3) and as a baseline to determine resulting attribute dependencies (Section 5). Rather than traversing the lattice using breadth-first search (as in Apriori) or depth-first search (as in FP-Growth) over the full domain, the proposed algorithm uses these violations to maintain an incrementally growing set of explored predicate sets, denoted by 𝑁 . For every explored predicate set 𝑃 ′ ∈ 𝑁 , the algorithm stores the parameters of its probability distribution, 𝛼 (𝑃 ′ ) and 𝛽 (𝑃 ′ ), inferred from evaluating all elements in its current sample Ω(𝑃 ′ ). Initially, the explored set contains only the root, all singletons, and all crossattribute pairs,  𝑁 = {∅} ∪ {𝑃𝑖 } : 𝑃𝑖 ∈ 𝑃  ∪ {𝑃𝑖 , 𝑃 𝑗 } : attr(𝑃𝑖 ) ≠ attr(𝑃 𝑗 ) ; The explored set is then iteratively updated through two stages: • Distribution refinement (Section 4.2). For every explored predicate set 𝑃 ′ ∈ 𝑁 , its sample is expanded, Ω(𝑃 ′ ) ← Ω(𝑃 ′ ) ∪ ΔΩ(𝑃 ′ ), and the corresponding parameters 𝛼 (𝑃 ′ ) and 𝛽 (𝑃 ′ ) are updated, reducing the uncertainty of the estimated probability. • Lattice expansion (Section 4.3). The boundary of the explored lattice is enlarged whenever the refined distributions provide sufficient statistical evidence that unexplored neighboring predicate sets may violate Equation 2.

4.2

Lattice Expansion

Given the current explored set 𝑁 together with the posterior parameters of every explored predicate set, the goal of this stage is to expand 𝑁 ← 𝑁 ∪ Δ𝑁 . The primary challenge is to prune the exponentially large search space without discarding predicate sets that may reveal meaningful dependencies. At any iteration, the posterior distributions provide sufficient information to determine whether there is statistical evidence that a predicate set violates Equation 2. Let 𝑆 ⊆ 𝑁 denote the collection of explored predicate sets containing a comparison whose soundness reaches 𝑠 0 . Recall from Definition 1 that such a violating set relates every pair of attributes represented within it. The expansion set Δ𝑁 is constructed by taking each explored predicate set 𝑃 ′ ∈ 𝑁 and adding one predicate from 𝑃 (𝐴𝑖 ), for 𝐴𝑖 ∉ attrs(𝑃 ′ ), whenever attribute 𝐴𝑖 is related to at least one attribute already represented in 𝑃 ′ . Intuitively, supersets obtained by introducing predicates from attributes that have never exhibited any statistical relationship with the attributes already present in 𝑃 ′ are not explored. Since combinations of independent attributes constitute the overwhelming majority of the lattice, this pruning strategy dramatically reduces the search space while ensuring that any discovered dependency can trigger exploration of the corresponding region of the lattice. Assumption 1 (Subset Traceability). Every multivariate dependency of interest exhibits, on at least √ one of its proper subsets, a conditional shift of at least 𝛿 min = 𝑠 0 2𝜖. This assumes that multivariate dependencies leave a statistical trail in their smaller subsets to guide the search. In the running example (Figure 2), flood-prone districts are slightly cheaper on average, and ground floors slightly discounted overall. Such weak marginal signals, once accepted, trigger exploration of the joint region where the dependence is strong. A traditional feature selector

Distribution Refinement

Given the current explored set 𝑁 , where each predicate set 𝑃 ′ ∈ 𝑁 has posterior parameters 𝛼 (𝑃 ′ ) and 𝛽 (𝑃 ′ ) inferred from sample

3 The sample size starts at 1000 elements and increases exponentially up to a maximum

of 106 in our implementation. This allows the algorithm to quickly identify distributions that converge early while avoiding unnecessary computation. 4 There are |𝑃 ′ |! paths from the root to node 𝑃 ′ . The algorithm uses the inferred distributions to estimate the optimal path parent of each node.

2 Φ −1 denotes the inverse cumulative normal distribution. A violation indicates the

two distributions would fail a statistical equality test of 95% confidence.

7

𝐶1

might not consider the weak pairwise conditional dependencies worth keeping, thus missing the stronger multivariate interaction. Consequently, DIADA implements a scalable heuristic rather than an exhaustive search: dependencies whose subsets carry no detectable signal (e.g., purely interactive ones) may be missed. Its pairwise guarantee is unaffected, as all cross-attribute pairs are explored by construction.

4.4

area_sqm

school_ quality

monthly_temp

monthly_ precipitations

market_ value

tax_rate

monthly_ humidity

monthly_ wind_speed

𝑠=5.1 floor_ number

𝑠≈82

𝑠=6.7

no edge crosses the spurious join property_id ≈ station_id

flood_risk

𝐶3

Termination and cost

Upon termination, every explored predicate set is associated with a posterior probability distribution whose uncertainty is bounded by the user-specified approximation factor 𝜖. These posterior distributions are then used to identify predicate sets that violate the conditional independence criterion of Equation 2, as described previously. Consequently, the computational cost is governed primarily by the uncertainty bound 𝜖 and the number of dependencies present in the data, rather than by exhaustive traversal of the entire lattice, making the approach scalable to large, high-dimensional datasets.

5

𝐶2 num_rooms

𝐶4 wall_color

wall_color_hex

property_id

postal_code

locker_number

station_id

each 𝐶𝑖 is materialized as one relation keys: excluded from 𝐺 , replicated into every 𝐶𝑖

Figure 3: The association graph 𝐺 for the running example of Figure 1. Edges join attributes co-occurring in some violating predicate set. We indicate the soundness of relationships breaking the independence hypothesis in Figure 2, following the posteriors at 𝜖 = 10−5 . The red edge carries the compari√ son 𝑝 (𝑃𝑉 | 𝑃𝐹 , 𝑃𝑅 ) vs. 𝑝 (𝑃𝑉 | 𝑃𝑅 ), |.600−.232|/ 2𝜖 ≈ 82. The four connected components are the composition C and become the four relations of Figure 1c.

THE DIADA SYSTEM

In this section, we present DIADA (Discovering Intrinsic Attribute Dependencies At scale) and its implementation details. For each relation in the input data lake, DIADA constructs a predicate space over its schema, applies the mining algorithm of Section 4 to obtain the set of statistically meaningful attribute dependencies (𝑅), and transforms 𝑅 into the relations of the composed layer.

𝐴𝑖 , the generated predicate set is fixed: ( {= 𝐴𝑖 , ≠ 𝐴𝑖 , < 𝐴𝑖 , > 𝐴𝑖 }, if 𝐴𝑖 has an ordered domain, 𝑃 (𝐴𝑖 ) = {= 𝐴𝑖 , ≠ 𝐴𝑖 }, otherwise. Although the resulting predicate spaces are less expressive than those obtained by enumerating all distinct attribute values, they provide a practical representation for identifying relationships between attributes. By capturing only the fundamental comparison operators, they provide a compact yet informative representation of attribute behavior, substantially reducing the search space while preserving the information required for dependency discovery.

Input. DIADA takes as input a wide relation, typically obtained by joining a base relation with candidate datasets returned by a discovery system [19, 34, 38]. As argued in Section 2, discovery establishes that datasets can be combined, but not whether their attributes should belong in the same relation, a gap that DIADA addresses. Therefore, DIADA is agnostic to the discovery algorithm and join conditions used. Nonetheless, recall that predicate evaluation is modeled as a Bernoulli trial over a uniformly sampled domain Ω, with the sampling distribution matching the one under evaluation. Thus, we fixe Ω = 𝑟 (𝑅𝑏 ) and require every candidate to be folded in without altering that domain. This requirement is violated by one-to-many joins, as a base tuple matching 𝑚 candidate tuples appears 𝑚 times in the result, contributing 𝑚 draws instead of one and reweighting the empirical distribution by match multiplicity. The resulting dependencies are indistinguishable from genuine data dependencies within the predicate space, invalidating Proposition 1. Each candidate should therefore be pre-aggregated by grouping on its join attribute before being combined via a multi-way left join, keeping |Ω| = |𝑟 (𝑅𝑏 )| regardless of candidate cardinality.

Attribute Dependencies. The system employs the algorithm described in Section 4 to infer joint predicate probability distributions whose posterior variance is bounded by the user-specified approximation factor 𝜖. These distributions are subsequently used to identify the collection of predicate sets 𝑃 ′ ⊂ 𝑃 that violate the conditional independence criterion introduced in Section 3. Rather than directly reporting the attribute sets corresponding to statistically significant predicate combinations, the system derives a pairwise dependency score for every pair of attributes. For any two attributes 𝐴𝑥 and 𝐴𝑦 , this score is defined as the maximum soundness observed among all pairs of conditional probabilities 𝑝 (𝑃𝑖 | 𝑃 𝑗 , 𝑃𝑘 , . . .) and 𝑝 (𝑃𝑖 | 𝑃𝑘 , . . .) violating Equation 2, where 𝑃𝑖 ∈ 𝑃 (𝐴𝑥 ) and 𝑃 𝑗 ∈ 𝑃 (𝐴𝑦 ). Intuitively, the score measures the greatest change in the behavior of a predicate derived from 𝐴𝑥 resulting from conditioning on a predicate derived from 𝐴𝑦 . Consequently, it quantifies the strongest statistical evidence that 𝐴𝑦 influences the behavior of 𝐴𝑥 , and can therefore be used directly as the test statistic for the independence hypothesis test described in Section 3.

Instantiating the predicate space. As described in Section 3, there are numerous ways of constructing predicate spaces. The most natural approach defines predicates over the domain of individual tuples, where each predicate is satisfied whenever a given attribute assumes a particular value or falls within a specified range (first row of Table 2). While this representation yields a highly expressive predicate space, its size grows proportionally with the number of distinct values present in the data, making it impractical for large and heterogeneous datasets. Instead, we construct predicates over the domain of tuple pairs (second row of Table 2). For each attribute

Compositional Components. These pairwise scores are turned into the association graph 𝐺, an undirected graph in which two columns are connected if the maximum soundness between  any con ditional predicate meets or exceeds a fixed threshold: Φ−1 1 − 0.01 |𝑃 ′ | , where Φ−1 denotes the inverse cumulative distribution function 8

Table 3: Characteristics of the considered datasets.

of the standard normal distribution and 𝑃 ′ is the set of predicates attributed to the score. This is the (1 − 0.01/|𝑃 ′ |) quantile of the standard normal distribution (i.e., roughly two standard deviations, applying a Bonferroni correction across the predicates entering each score). Composing then reduces to traversing the constructed graph and defining each connected component as a relation. For example, given attributes {𝐴, 𝐵, 𝐶} ∈ 𝑅, if 𝐴 relates to 𝐵 and 𝐵 relates to 𝐶, then 𝐴, 𝐵, and 𝐶 are placed in the same component because, even if 𝐴 and 𝐶 are not directly related, they have a semantic link through 𝐵. Columns with no qualifying relationship are treated as univariate noise and collected into a single component rather than discarded outright. Columns that a join was actually performed on are propagated into every resulting relation, so each table remains independently joinable back via a shared key. Figure 3 shows the compositional components generated for the example in Figure 1.

6

EXPERIMENTS

Our evaluation answers four questions, tied to the requirements defined in Section 2: RQ1 Does the mining algorithm scale in rows and columns beyond the classical lattice miners? (R4) RQ2 Does composition separate noise from signal better than unsupervised feature selection? (R1–R3) RQ3 Does composing the data before modeling improve downstream ML pipelines? (R2–R3) RQ4 Does a single, untargeted composition support inference over several targets defined only afterwards? (R1, R3) Benchmarks. We collect 17 public datasets spanning binary, multiclass, and regression tasks (Table 3), and derive from each two noisy benchmarks by adding columns, emulating the result of spurious joins. The univariate noise (UN) benchmark adds, for every original non-target column, two columns of pure noise (one numerical, one categorical, with varying distributions) and two shuffled copies of the column itself, preserving its distribution while destroying every relationship. The multivariate noise (MN) benchmark adds, on top of UN, a cluster of synthetic features correlated with one another but with nothing else, emulating an entire relation spuriously merged. This results in benchmarks ranging from 41 to 406 columns. These benchmarks mimic Figure 1b: discovery produces a large dataset mixing related attributes (base datasets) with uninformative ones (all the generated columns), and the task is to recover the meaningful relations. UN requires removing the noise while keeping the original dataset, whereas MN additionally requires isolating the intra-correlated cluster. Injecting synthetic noise gives an unambiguous ground truth stating which columns must not be kept, letting us measure how much composition benefits downstream analysis. Nonetheless, we acknowledge the need to validate on naturally occurring heterogeneous lakes, which is left for future work. In particular, uncontrolled compositions in such repositories might combine all datasets into a single component. Even if supported by our statistical criterion, such a result hinders downstream usage. 𝜖 is set to its default value of 10−5 for Sections 6.2–6.4.

6.1

Dataset

Task

#Rows

#Columns

Bank Default Payment Jannis Miniboone Covertype Drive Diagnosis Dry Beans Mice Protein Pendigits Students Appliances Avocado Sales Diamonds House Sales Nasa Pol Superconductivity

Binary Binary Binary Binary Multiclassification (7) Multiclassification (10) Multiclassification (7) Multiclassification (8) Multiclassification (10) Multiclassification (3) Regression Regression Regression Regression Regression Regression Regression

45k 30k 57k 73k 40k 58k 13k 1k 11k 5k 20k 18k 54k 21k 45k 15k 21k

16 24 54 50 13 48 17 81 17 36 28 13 9 17 22 48 82

for which Apriori [2] and FP-Growth [22] represent the canonical mining algorithms. We compare DIADA to them in order to isolate the effect of our pruning rather than as multivariate discovery competitors, given that, as argued in Section 2.4, no existing method mines multivariate dependencies at lake scale. Thus, the column scalability comparison below doubles as a pruning ablation: the exhaustive miners enumerate the lattice while DIADA expands it only under statistical evidence. We analyze the three algorithms on the two scalability axes introduced in Section 4 (rows and columns) and report results on three of the largest benchmarks, jannis, miniboone and drive_diagnosis (Figure 4). To ensure a fair comparison that isolates the lattice exploration strategies from the underlying evaluation mechanisms, we adapt Apriori and FP-Growth to operate on a sampled domain Ω ′ ⊂ Ω. The sample size |Ω ′ | is determined directly by 𝜖 and is chosen as the minimum value that guarantees the uncertainty of every explored lattice node remains bounded by 𝜖.5 This represents a best-case scenario for applying these algorithms to derive predicate distributions with bounded uncertainty, providing a fair basis for comparison. Thus, Figure 4 reports the execution time each algorithm requires to guarantee that the variance of every explored lattice node stays within the prescribed value of 𝜖. Row Scalability. Row scalability refers to the capacity of the algorithms to deal with larger volumes of data, determined by 𝜖. Figure 4 shows FP-Growth’s advantage over Apriori growing with sample size: larger samples add redundancy that FP-Growth’s compressed data structures exploit effectively. For highly dimensional datasets, however, these structures eventually exceed practical memory limits. Our algorithm offers an intermediate alternative, matching FPGrowth’s performance while avoiding the construction of expensive 5 For a fixed sample size 𝑛 , the probability distribution inferred from 𝑎 successes and 𝑏

failures, where 𝑎 + 𝑏 = 𝑛 , attains its maximum variance at the boundary cases (𝑎 = 0 or 𝑏 = 0). In these cases, the variance is proportional to 𝑛1 . Consequently, by selecting 𝑛 = 𝑘𝜖 , there exists a numerical constant 𝑘 such that every inferred distribution has variance bounded by 𝜖 . For the statistical tests considered in this work, numerical evaluation yields 𝑘 = 5.

Scalability of the mining algorithm

The predicate abstraction reduces multivariate dependency discovery to the problem of frequent-itemset and association-rule mining, 9

Table 4: Analysis of composition quality. We report the average (± 95% CI) percentage of noise columns, base columns and targets kept across all benchmarks.

Figure 4: Row/column scalability of DIADA, Apriori and FPGrowth for estimating joint predicate probabilities of lattice predicate sets. Executions are repeated 5 times, each time evaluating on different random sets of columns, and the average is reported. Missing entries indicate memory overflows.

Noise (%)

Original (%)

Target (%)

DIADA Cluster Dispersion [15] PFA [31] RSR [55] MCFS [9] UDFS [51] SPEC [54] NDFS [29] Cluster Representatives Laplacian Score [23] AGUFS [24] AutoEncoder [4] Variance Score InfFS [45] Correlation Uniqueness

1.6 ± 2.3 4.4 ± 6.8 5.3 ± 5.4 8.2 ± 9.3 11.1 ± 13.8 17.2 ± 10.6 20.6 ± 14.7 24.5 ± 21.0 35.3 ± 7.7 44.2 ± 15.0 62.6 ± 8.3 65.8 ± 6.2 66.7 ± 0.3 69.1 ± 14.0 90.2 ± 8.3

91.5 ± 9.9 27.3 ± 12.9 40.3 ± 18.1 5.0 ± 3.1 12.5 ± 13.8 34.2 ± 19.8 3.6 ± 2.4 3.4 ± 2.9 13.7 ± 3.9 81.5 ± 16.1 87.8 ± 13.9 13.9 ± 5.0 81.9 ± 8.3 25.6 ± 19.8 7.3 ± 8.2

100.0 23.5 47.1 0.0 17.6 41.2 5.9 5.9 5.9 88.2 82.4 23.5 82.4 35.3 0.0

The two numbers must be read jointly: Cluster Dispersion or PFA keep noise comparatively low, but only by being so restrictive that most of the signal is discarded with it, whereas methods that retain most signal (e.g., Laplacian Score or AGUFS) admit 44–67% noise. DIADA is also the only method that never discards the prediction target. Thus, UFS techniques are failing to preserve the only feature whose meaningfulness can be guaranteed. Two causes explain this behavior. First, UFS approaches assume the datasets’ structure is worth preserving, so when the input’s structure is contaminated by noise, the objective protects exactly what should be removed. Second, excessive parametrization: every method requires an assortment of hyperparameters with no principled criterion to set them, so performance varies considerably across benchmarks, as the wide confidence intervals showcase. DIADA avoids both problems by employing a statistical approach to assess multivariate attribute dependencies and using a single tuning parameter. As a result of preserving too much noise and removing too many meaningful attributes, UFS methods are not suitable for composition tasks and thus we only analyze DIADA further.

global compression structures. Instead, it concentrates computation on statistically relevant regions of the search space without requiring global data structures that summarize the entire domain. Column Scalability. Column scalability is particularly important in data lakes, where joins across multiple sources produce relations with hundreds of attributes. The advantage of our exploration strategy is evident here. Whereas Apriori and FP-Growth systematically explore predicate combinations that are often mutually independent, our method postpones exploration until there is statistical evidence suggesting dependence. This selective exploration substantially reduces the number of lattice nodes visited, mitigating the exponential growth inherent to exhaustive search. Importantly, this is achieved without imposing an explicit upper bound on predicate size: the algorithm is guaranteed by construction to discover all pairwise dependencies while continuing to explore deeper levels of the lattice whenever the observed evidence warrants it.

6.2

Method

Separating signal from noise

DIADA and UFS methods are the only alternatives that meet the composition requirements defined in Section 2. Hence, we first evaluate their capacity to preserve meaningful features in the UN benchmarks. As building a ground truth from exact multivariate dependencies is unfeasible, we define meaningful features as those originally present in the benchmarks. These are popular prediction datasets, so dependencies between the target and a large subset of the attributes are assumed to be present by construction. Thus, the goal is to remove the injected noise columns while keeping the original attributes and the target. We run DIADA and 14 popular UFS methods, classical and modern and with varied selection objectives, on all 17 benchmarks. Table 4 presents the averaged results for the amount of selected noise columns, base attributes and targets selected across benchmarks. DIADA selects the lowest percentage of noise columns (1.6%) while retaining the largest amount of original attributes (91.5%).

6.3

Improving downstream pipelines

The main goal of composition is to provide a series of meaningful relations that can benefit the performance of downstream tasks. In this experiment, we compare the results of ML modeling pipelines on the designed benchmarks (UN and MN) before (pre-) and after (post-) composition. To do so, we have defined a two-step modeling pipeline including a data augmentation system, ARDA [11], to rank features by inferred relevance, and AutoML to train the models. Hence, we simulate a large-scale pipeline requiring automated processing due to the infeasibility of manually assessing feature relevance. We compare three inputs: (1) pre-composition, where the UN and MN benchmarks reach the pipeline directly (i.e., the output of automated data discovery); (2) post-composition, where the UN and MN benchmarks have been composed and, thus, only meaningful attributes are fed to the ML pipeline; and (3) base, which represents 10

Figure 5: Model utility (top) and share of noise features selected (bottom) as ARDA selects more features, on representative benchmarks for both classification and regression tasks. Post-composition lines stop at the width of the composed relation. the original datasets without any noise, serving as a baseline for the expected models’ behavior. Note that for UN benchmarks the models ingest attributes in all found components, whereas for MN they ingest only the component containing the target variable. This showcases the difference between only preserving statistically dependent features and further composing the data into components. We evaluate model utility for an increasing amount of features selected (𝑘), from 1 up to the minimum between the original width and 40. For representative benchmarks (Figure 5), we track model utility (top panels) and the percentage of noise columns selected (bottom panels). The remaining datasets, in our artifact repository, showcase the same trends. The pre-composition results show that without composition downstream tasks are unable to filter noise. Several benchmarks (those with a difficult target to predict) end up selecting up to 60% of noisy features. This is a limitation of scalable methods to select features, as they rely on cheap, high-level proxies to assess relevance rather than on complex, but accurate techniques. This triggers widely studied issues in model creation (e.g., true predictors being collectively mimicked by noise variables when the marginal target signal is weak [16, 17]), so the resulting models rest on spurious associations. Even at high utility they generalize poorly under distribution shift, as the captured patterns do not describe replicable phenomena. Composing first almost eliminates the problem. Consistently with Section 6.2, post-composition models are built almost exclusively from original features, with the only notable exception being students, likely due to the poor quality of its attributes that facilitates the generation of relationships between noise and original features. The utility for post-composition models is never worse

than pre-composition models (e.g., diamonds, nasa) and often considerably better (e.g., default payment, house sales, students). Moreover, the UN–MN gap on post-composition models is generally minimal, confirming that the injected intra-correlated cluster is isolated and removed from modeling. In fact, some MN compositions keep fewer original features than the UN ones (e.g., bank, default_payment) at no cost in utility, implying the existence of attribute groups of the original data that are internally related but independent of the target’s relation. These are correctly filtered out. This is only visible in the MN benchmarks, where the ML model is fed solely the component that has the target. The only failed model is appliances for the MN benchmark, whose composition leaves a single companion attribute in the target’s relation, crippling the model. This is the result of the small percentage of error tolerated by statistical approaches, which can be addressed by lowering 𝜖. Overall, composition increases model performance, robustness and favors the detection of real patterns. Against base, post-composition models perform, generally, to an equal degree, indicating that DIADA removes all the noise that negatively impacts the model. Moreover, post-composition models perform better in some cases. On jannis both compositions retain 26 of 54 features, and the model reaches 0.8 accuracy around 20 selected features whereas base needs 26, with no gains beyond. Thus, composition is removing original features that provide no inference benefit without having to rely on pairwise comparison with the target variable. pol behaves similarly, and default payment and house sales do so to a lesser extent. The effect is not universal (superconductivity and nasa retain their full width), indicating either genuinely meaningful attributes (although redundant for the model) or redundancy that DIADA does not capture. 11

Figure 6: Model utility (top of each panel) and share of noise features selected (bottom) for two synthetic targets defined after composition, on the representative benchmarks of Figure 5. Remaining benchmarks in the artifact repository.

6.4

One composition, any target

conditioning on specific targets. We propose an information-driven composition approach that approximates multivariate dependence by identifying sets of attributes that break the independence hypothesis. We map the attributes to a predicate space, which serves as an abstraction layer that decouples multivariate dependency analysis from the data’s underlying types. The predicates form a lattice under inclusion, and we contribute a dedicated and scalable algorithm that mines dependent predicate sets. The algorithm is instantiated in DIADA, the first system to fully satisfy all four data composition requirements (R1–R4). DIADA is able to efficiently scale beyond classical mining algorithms, thus allowing the detection of dependencies that would otherwise be impractical, and the target-agnostic compositions it produces benefit diverse downstream tasks by providing a subset of low-noise, statistically meaningful attributes. Future work aims to solve the loss of information due to predicate abstraction, add safeguards to prevent composition collapsing into a single component, and extend DIADA to heterogeneous domains and predicate spaces other than tuple pairs.

To test whether one composition can serve several downstream tasks we generate two synthetic targets for each of the 17 UN benchmarks, apply a single composition process and check both models’ behavior. The synthetic targets have been generated through a latent score drawn as a random linear combination over one half of the attributes for the first target and the other half for the second. This score is rank-transformed to [0, 1] for robustness and mapped to the appropriate task (e.g. sigmoid-thresholded for binary labels). Synthetic targets are, thus, learnable from the source attributes while distinct from the original labels. We repeat the protocol of Section 6.3, now for both targets, presenting pre-composition, postcomposition and base results in Figure 6. As in Section 6.3, post-composition models perform close to base models for both targets across most benchmarks, and, again, better than pre-composition models, as the only noticeable exception is the second target of jannis, which again exemplifies the small rate of error associated with statistical processes. Once more, noise is almost eradicated from the relations, thus favoring patterns that generalize to unseen data and not based on spurious associations. The composition was executed without knowledge of either target, implying that DIADA selects the attributes that characterize the relation, not the ones that explain a designated variable. Hence, a single composition process generates relations that can serve meaningful attributes to any subsequent analytical task.

7

ACKNOWLEDGMENTS This work has been partly supported by the Horizon Europe Programme under GA.101135513 (CyclOps) and the Spanish Ministerio de Ciencia e Innovación under project PID2023-152841OA-I00 / AEI/10.13039/501100011033 (TALC). Marc Maynou is supported by Google’s PhD Fellowship Program. Anna Queralt is a Serra Húnter Fellow. Albert Martin is funded by the predoctoral program AGAUR-FI grants (2025 FI-1 00967) Joan Oró, which is backed by the Department of Research and Universities of the Generalitat of Catalonia, as well as the European Social Plus Fund.

CONCLUSION

We introduced the problem of data composition: organizing a fragmented, heterogeneous data lake into meaningful relations without 12

REFERENCES

[28] Batya Kenig, Pranay Mundra, Guna Prasaad, Babak Salimi, and Dan Suciu. 2020. Mining Approximate Acyclic Schemes from Relations. In SIGMOD Conference. ACM, 297–312. [29] Zechao Li, Yi Yang, Jing Liu, Xiaofang Zhou, and Hanqing Lu. 2012. Unsupervised feature selection using nonnegative spectral analysis. In Proceedings of the AAAI conference on artificial intelligence, Vol. 26. 1026–1032. [30] Hyunki Lim and Dae-Won Kim. 2021. Pairwise dependence-based unsupervised feature selection. Pattern Recognition 111 (2021), 107663. [31] Yijuan Lu, Ira Cohen, Xiang Sean Zhou, and Qi Tian. 2007. Feature selection using principal feature analysis. In Proceedings of the 15th ACM international conference on Multimedia. 301–304. [32] Albert Martin, Eduardo C. De Almeida, Oscar Romero, and Anna Queralt. 2025. How and Why False Denial Constraints are Discovered. Proceedings of the VLDB Endowment 18, 10 (jun 2025), 3477–3489. [33] Albert Martin, Eduardo C. De Almeida, Oscar Romero, and Anna Queralt. 2026. Discovering Approximate Denial Constraints in Large Databases. Proceedings of the VLDB Endowment 19, 9 (july 2026), 2086 – 2098. [34] Marc Maynou, Sergi Nadal, Raquel Panadero, Javier Flores, Oscar Romero, and Anna Queralt. 2026. Freyja: Efficient Join Discovery in Data Lakes. IEEE Trans. Knowl. Data Eng. 38, 4 (2026), 2277–2288. [35] William J. McGill. 1954. Multivariate information transmission. Trans. IRE Prof. Group Inf. Theory 4 (1954), 93–111. https://doi.org/10.1109/TIT.1954.1057469 [36] Pabitra Mitra, Chivukula Anjaneya Murthy, and Sankar K. Pal. 2002. Unsupervised feature selection using feature similarity. IEEE transactions on pattern analysis and machine intelligence 24, 3 (2002), 301–312. [37] Fatemeh Nargesian, Ken Q. Pu, Bahar Ghadiri Bashardoost, Erkang Zhu, and Renée J. Miller. 2023. Data Lake Organization. IEEE Trans. Knowl. Data Eng. 35, 1 (2023), 237–250. [38] Fatemeh Nargesian, Erkang Zhu, Renée J Miller, Ken Q Pu, and Patricia C Arocena. 2019. Data lake management: challenges and opportunities. Proceedings of the VLDB Endowment 12, 12 (2019), 1986–1989. [39] Fatemeh Nargesian, Erkang Zhu, Ken Q Pu, and Renée J Miller. 2018. Table union search on open data. Proceedings of the VLDB Endowment 11, 7 (2018), 813–825. [40] Thorsten Papenbrock, Jens Ehrlich, Jannik Marten, Tommy Neubert, Jan-Peer Rudolph, Martin Schönberg, Jakob Zwiener, and Felix Naumann. 2015. Functional Dependency Discovery: An Experimental Evaluation of Seven Algorithms. Proc. VLDB Endow. 8, 10 (2015), 1082–1093. [41] Leonardo Pellegrina and Fabio Vandin. 2018. Efficient Mining of the Most Significant Patterns with Permutation Testing. In KDD. ACM, 2070–2079. [42] Eduardo HM Pena, Eduardo C De Almeida, and Felix Naumann. 2019. Discovery of approximate (and exact) denial constraints. Proceedings of the VLDB Endowment 13, 3 (2019), 266–278. [43] Matteo Riondato, Justin A. DeBrabant, Rodrigo Fonseca, and Eli Upfal. 2012. PARMA: a parallel randomized algorithm for approximate association rules mining in MapReduce. In CIKM. ACM, 85–94. [44] Matteo Riondato and Eli Upfal. 2014. Efficient discovery of association rules and frequent itemsets through sampling with tight performance guarantees. ACM Transactions on Knowledge Discovery from Data (TKDD) 8, 4 (2014), 1–32. [45] Giorgio Roffo, Simone Melzi, and Marco Cristani. 2015. Infinite feature selection. In Proceedings of the IEEE international conference on computer vision. 4202–4210. [46] Saúl Solorio-Fernández, J Ariel Carrasco-Ochoa, and José Fco Martínez-Trinidad. 2020. A review of unsupervised feature selection methods. Artificial intelligence review 53, 2 (2020), 907–948. [47] Hannu Toivonen. 1996. Sampling Large Databases for Association Rules. In VLDB. Morgan Kaufmann, 134–145. [48] Jorge R Vergara and Pablo A Estévez. 2014. A review of feature selection methods based on mutual information. Neural computing and applications 24, 1 (2014), 175–186. [49] Satosi Watanabe. 1960. Information theoretical analysis of multivariate correlation. IBM Journal of research and development 4, 1 (1960), 66–82. [50] Geoffrey I. Webb. 2007. Discovering Significant Patterns. Mach. Learn. 68, 1 (2007), 1–33. [51] Yi Yang, Heng Tao Shen, Zhigang Ma, Zi Huang, and Xiaofang Zhou. 2011. ℓ2,1 Norm Regularized Discriminative Feature Selection for Unsupervised Learning. In IJCAI international joint conference on artificial intelligence. 1589–1594. [52] Zhong Yuan, Hongmei Chen, Pengfei Zhang, Jihong Wan, and Tianrui Li. 2022. A novel unsupervised approach to heterogeneous feature selection based on fuzzy mutual information. IEEE Transactions on fuzzy systems 30, 9 (2022), 3395–3409. [53] Yunjia Zhang, Zhihan Guo, and Theodoros Rekatsinas. 2020. A Statistical Perspective on Discovering Functional Dependencies in Noisy Data. In SIGMOD Conference. ACM, 861–876. [54] Zheng Zhao and Huan Liu. 2007. Spectral feature selection for supervised and unsupervised learning. In Proceedings of the 24th international conference on Machine learning. 1151–1157. [55] Pengfei Zhu, Wangmeng Zuo, Lei Zhang, Qinghua Hu, and Simon CK Shiu. 2015. Unsupervised feature selection by regularized self-representation. Pattern Recognition 48, 2 (2015), 438–446.

[1] Daniel Abadi, Anastasia Ailamaki, David Andersen, Peter Bailis, Magdalena Balazinska, Philip Bernstein, Peter Boncz, Surajit Chaudhuri, Alvin Cheung, AnHai Doan, et al. 2020. The Seattle report on database research. ACM Sigmod Record 48, 4 (2020), 44–53. [2] Rakesh Agrawal, Tomasz Imielinski, and Arun N. Swami. 1993. Mining Association Rules between Sets of Items in Large Databases. In SIGMOD Conference. ACM Press, 207–216. [3] Michael Armbrust, Ali Ghodsi, Reynold Xin, Matei Zaharia, et al. 2021. Lakehouse: a new generation of open platforms that unify data warehousing and advanced analytics. In 11th Conference on Innovative Data Systems Research, CIDR 2021. [4] Muhammed Fatih Balın, Abubakar Abid, and James Zou. 2019. Concrete autoencoders: Differentiable feature selection and reconstruction. In International conference on machine learning. PMLR, 444–453. [5] Anthony J Bell. 2003. The co-information lattice. In Proceedings of the fifth international workshop on independent component analysis and blind signal separation: ICA, Vol. 2003. [6] Christopher M Bishop. 2006. Pattern recognition and machine learning. Springer. [7] Alex Bogatu, Alvaro AA Fernandes, Norman W Paton, and Nikolaos Konstantinou. 2020. Dataset discovery in data lakes. In 2020 ieee 36th international conference on data engineering (icde). IEEE, 709–720. [8] Gavin Brown, Adam Pocock, Ming-Jie Zhao, and Mikel Luján. 2012. Conditional likelihood maximisation: a unifying framework for information theoretic feature selection. The journal of machine learning research 13 (2012), 27–66. [9] Deng Cai, Chiyuan Zhang, and Xiaofei He. 2010. Unsupervised feature selection for multi-cluster data. In Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining. 333–342. [10] Riccardo Cappuzzo, Aimee Coelho, Félix Lefebvre, Paolo Papotti, and Gaël Varoquaux. 2025. Retrieve, Merge, Predict: Augmenting Tables with Data Lakes. Trans. Mach. Learn. Res. 2025 (2025). [11] Nadiia Chepurko, Ryan Marcus, Emanuel Zgraggen, Raul Castro Fernandez, Tim Kraska, and David R. Karger. 2020. ARDA: Automatic Relational Data Augmentation for Machine Learning. Proc. VLDB Endow. 13, 9 (2020), 1373–1387. [12] Xu Chu, Ihab F Ilyas, and Paolo Papotti. 2013. Discovering denial constraints. Proceedings of the VLDB Endowment 6, 13 (2013), 1498–1509. [13] Lingxi Cui, Huan Li, Ke Chen, Lidan Shou, and Gang Chen. 2026. Tabular data augmentation for machine learning: Progress and prospects of embracing generative ai. Comput. Surveys 58, 12 (2026), 1–39. [14] Databricks. 2023. What Is the Medallion Lakehouse Architecture? https://www. databricks.com/glossary/medallion-architecture [15] Jennifer G Dy and Carla E Brodley. 2004. Feature selection for unsupervised learning. Journal of machine learning research 5, Aug (2004), 845–889. [16] Jianqing Fan and Jinchi Lv. 2008. Sure independence screening for ultrahigh dimensional feature space. Journal of the Royal Statistical Society Series B: Statistical Methodology 70, 5 (2008), 849–911. [17] Jianqing Fan, Qi-Man Shao, and Wen-Xin Zhou. 2018. Are discoveries spurious? Distributions of maximum spurious correlations and their applications. Annals of statistics 46, 3 (2018), 989–1018. [18] Raul Castro Fernandez, Ziawasch Abedjan, Famien Koko, Gina Yuan, Samuel Madden, and Michael Stonebraker. 2018. Aurum: A Data Discovery System. In ICDE. IEEE Computer Society, 1001–1012. [19] Raul Castro Fernandez, Essam Mansour, Abdulhakim A Qahtan, Ahmed Elmagarmid, Ihab Ilyas, Samuel Madden, Mourad Ouzzani, Michael Stonebraker, and Nan Tang. 2018. Seeping semantics: Linking datasets using word embeddings for data discovery. In 2018 IEEE 34th International Conference on Data Engineering (ICDE). IEEE, 989–1000. [20] Isabelle Guyon and André Elisseeff. 2003. An introduction to variable and feature selection. Journal of machine learning research 3, Mar (2003), 1157–1182. [21] Rihan Hai, Christos Koutras, Christoph Quix, and Matthias Jarke. 2023. Data lakes: A survey of functions and systems. IEEE Transactions on Knowledge and Data Engineering 35, 12 (2023), 12571–12590. [22] Jiawei Han, Jian Pei, and Yiwen Yin. 2000. Mining Frequent Patterns without Candidate Generation. In SIGMOD Conference. ACM, 1–12. [23] Xiaofei He, Deng Cai, and Partha Niyogi. 2005. Laplacian score for feature selection. Advances in neural information processing systems 18 (2005). [24] Yanyong Huang, Zongxin Shen, Fuxu Cai, Tianrui Li, and Fengmao Lv. 2021. Adaptive graph-based generalized regression model for unsupervised feature selection. Knowledge-Based Systems 227 (2021), 107156. [25] Yka Huhtala, Juha Kärkkäinen, Pasi Porkka, and Hannu Toivonen. 1999. TANE: An efficient algorithm for discovering functional and approximate dependencies. The computer journal 42, 2 (1999), 100–111. [26] Ihab F. Ilyas, Volker Markl, Peter J. Haas, Paul Brown, and Ashraf Aboulnaga. 2004. CORDS: Automatic Discovery of Correlations and Soft Functional Dependencies. In SIGMOD Conference. ACM, 647–658. [27] Andra Ionescu, Kiril Vasilev, Florena Buse, Rihan Hai, and Asterios Katsifodimos. 2024. AutoFeat: Transitive Feature Discovery over Join Paths. In ICDE. IEEE, 1861–1873. 13

Related documents

• DIADA: Automatic Data Composition in Data Lakes#1156917
• Data Descriptive Language for Shared Data#137099
• Ontological Deep Data Cleaning#387051
• An ontological investigation over human relations in linked data#774434
• CoGe-GCD: Reframing Generalized Category Discovery with Compositional Generalization#997069