Universal Encoders for Modular Relational Deep Learning Jakub Peleška1
and Gustav Šír1
arXiv:2606.21434v1 [cs.LG] 19 Jun 2026
Czech Technical University in Prague, Karlovo náměstí 13, Prague, 121 35, Czechia {jakub.peleska,gustav.sir}@cvut.cz
Abstract. Relational Deep Learning (RDL) models multi-tabular databases as temporal heterogeneous graphs for end-to-end representation learning. While RDL is evolving rapidly, existing approaches face significant generalization obstacles. They are either schema-specific, requiring training from scratch for every new database, or they rely on monolithic architectures that entangle feature encoding with graph message-passing. Analyzing these limitations, we establish four core pillars for building foundational relational models: semantic granularity, structural topology, temporal causality, and unified optimization. Addressing these pillars, we propose a modular approach that decouples row encoding from graph message-passing. For that purpose, we introduce the Universal Row Encoder - a transformer-based module that integrates raw cell data with schema metadata—including column semantics, table names, and global distribution statistics—to produce table-width invariant row embeddings. By explicitly feeding global statistics to an intra-row self-attention mechanism, the encoder natively contextualizes unseen features and handles sparse data. Serving as a flexible “backend” for any downstream graph architecture, our pretrained encoder enhances cross-database knowledge transfer on the established RelBench benchmarks while improving learning convergence and memory footprint. Keywords: Relational Deep Learning · Foundational Models · Relational Databases · Pretraining
1
Introduction
Traditional machine learning pipelines for relational databases (DBs) typically require extensive, manual feature engineering to flatten interconnected tables into a single, static data matrix. In recent years, Relational Deep Learning (RDL) [3,35,5] has emerged as a powerful alternative, modeling multi-tabular databases natively as temporal heterogeneous graphs. By representing rows as nodes and primary-foreign key linkages as edges, RDL frameworks operate directly on the raw relational structure by leveraging Heterogeneous Graph Neural Network (HGNN) architectures [36]. This shift allows for end-to-end representation learning that inherently respects the complex relational topology, underlying the DBs.
2
J. Peleška and G. Šír
However, despite its rapid evolution, most of RDL is still constrained by a severe generalization bottleneck. Standard HGNNs, utilized for RDL, are considerably schema-specific because the input layers and message-passing mechanisms of these models are hardcoded to the specific tables and columns of the training DB; hence, they cannot easily adapt to new databases or new domains. A model trained on a retail database, for instance, cannot be directly transferred to a healthcare database due to disjoint feature spaces and relational structures. Consequently, practitioners are forced to initialize and train a new model from scratch for every new database, limiting the scalability and adoption of RDL. To overcome these limitations, the RDL community needs to transition towards relational foundation models. The overarching goal is to leverage supervised pretraining across a massive, diverse corpus of multi-domain databases to learn universal representations of tabular data and structural motifs. By capturing these broad patterns, a foundational RDL model promises the ability to achieve zero-shot or few-shot transfer on entirely unseen schemas, dramatically reducing both the computational overhead and the volume of labeled data required for new predictive tasks. While there have been some very recent proposals in this direction [31,30,24], these current works largely mirror other AI domains by advocating monolithic Transformer-based architectures that entangle feature encoding with graph message passing, blurring the boundary between attribute semantics and relational structure, weakening the key inductive bias salient to the relational DB domain and robust schema transfer. In this work, we present a two-fold contribution to address these architectural constraints. First, we establish a conceptual framework formalized as the Four Pillars of foundational relational modeling: semantic granularity, structural topologies, temporal causality, and unified optimization. These pillars dictate the necessity of a modular pretraining framework that explicitly decouples feature encoding from relational message-passing. Second, as our core technical contribution focusing directly on the first pillar, we introduce the Universal Row Encoder. This transformer-based module is designed to produce rich, table-width invariant embeddings. By integrating raw cell data with crucial schema metadata—such as column semantics, global distribution statistics, and table names—the encoder natively contextualizes unseen features and robustly handles missing data without flattening the entire database into a global context bottleneck. Crucially, this modular design provides significant architectural flexibility while retaining the original relational learning bias. The pretrained Universal Row Encoder acts as a robust, plug-and-play module for any downstream graph architecture (e.g., Heterogeneous Graph Transformer [10] or GraphSAGE [8]) and pairs seamlessly with diverse decoders, ranging from task-specific multi-layer perceptrons to unified In-Context Learning (ICL) [1] modules. The Universal Row Encoder has been implemented as a component of the ReDeLEx framework [21], and its source code is readily available on GitHub1 .
1
https://github.com/jakubpeleska/redelex
Universal Encoders for Modular Relational Deep Learning
3
Zero-Shot Transfer & Schema Generalization
Relational Foundation Model 1. Semantic Granularity
2. Structural Topologies Node
3. Temporal Causality t
4. Unified Optimization
xf ut
Node
× Hub
Row Encoder
Txt
Num
Time
Node
ttgt
Node
L
xtgt
xpast
Cls.
Regr.
Link.
Raw Multi-Tabular Databases Fig. 1. The Four Pillars of Relational Foundation Models. A modular architecture is required to systematically resolve semantic granularity, structural topologies, temporal causality, and unified optimization without entangling the representational space.
2
The Pillars of the Relational Foundation Models
To successfully pretrain a foundational RDL model that generalizes across diverse databases, we propose mapping the fundamental properties of relational data. We establish four core “pillars” of relational modeling (visualized in Figure 1). These pillars introduce complexities that cannot be directly resolved by monolithic architectures or existing flattening (propositionalization) techniques [15]. Attempting to force these challenges into a single, entangled learning representation inevitably leads to memory bottlenecks, temporal leakage, or loss of transferability. Instead, resolving these pillars motivates a modular approach, in which specialized components handle specific complexities before integrating into a unified architecture. 2.1
Semantic & Feature Granularity
Cross-Table Feature Distribution Databases contain diverse data types (continuous, categorical, text, timestamps) with different scales. A foundational model must map these schema-specific signals into a shared, universal latent space without relying on hardcoded input dimensions [11]. Intra-Table Feature Relationships Before an entity can share information across a graph, its own local context must be understood. Intra-row relationships are semantic (e.g., “price” and “discount” must be contextualized together). Architectures that treat every cell as a distinct node (token) in a massive global graph context can dilute these high-density intra-row relationships. Robustness to Sparse Data Real-world relational DBs are rarely clean; they are sparse and frequently contain missing values. A foundational model requires
4
J. Peleška and G. Šír
objectives that teach it to impute missing values and reason over incomplete relational contexts, which calls for dedicated feature-level processing mechanisms.
2.2
Structural & Relational Topologies
The Semantics of PK-FK Relationships Not all edges in a relational graph are equivalent. A Primary Key-Foreign Key (PK-FK) linkage might represent a “belongs-to” relationship or a “happened-at” relationship. The model must distinguish between these relational semantics to route information correctly.
Overcoming Structural Bottlenecks Relational schemas often form “star” schemas, creating massive hub nodes. Standard message-passing GNNs can suffer from over-smoothing and neighborhood explosion when traversing these hubs [8]. Foundational models require structural routing to navigate such bottlenecks without losing signal fidelity.
Strict Schema-Agnosticism The ultimate goal of a foundation model is transferability. The architecture must be agnostic to the graph topology and node types of the pretraining data, enabling it to ingest novel relational structures during inference without requiring structural retraining.
2.3
Temporal Causality
Continuous Time Representation and Strict Causality Most real-world DBs are living records of events over time. Treating time as a continuous, causal dimension is critical to prevent temporal leakage during message passing. Monolithic entangled spatio-temporal graphs easily violate causality unless temporal constraints are decoupled and enforced.
Asynchronous Multi-Table Dynamics In real-world databases, temporal events occur at vastly different frequencies across different tables. A users table might update rarely, while a clicks or transactions table generates thousands of rows per second. A foundational model must be able to align these asynchronous timelines, avoiding the over-representation of high-frequency tables while retaining crucial low-frequency state changes.
Temporal Concept Drift and Feature Evolution Entities in a database are not static; their behaviors and data distributions shift over time (concept drift) [6]. A foundational model must learn temporally adaptive representations. It requires mechanisms to decay outdated information and weigh recent context appropriately, rather than treating history as a flat set of static edges.
Universal Encoders for Modular Relational Deep Learning
5
Modular RDL Framework Arbitrary Database with Prediction Task
Universal Row Encoder
Uniform Node Features ∈ Rd
Graph Neural Architecture
Contextualized Embeddings
Prediction Head
Output Prediction
Fig. 2. The modular framework for Relational Deep Learning. The architecture explicitly decouples semantic feature extraction from structural message passing. The Universal Row Encoder standardizes arbitrary database rows into uniform node features, which are then processed by a graph neural network, and finally mapped to predictions via task-specific decoders.
2.4
Unified Optimization & Transferability
Unified Task Representation Supervised pretraining on relational data requires learning from vastly different tasks (e.g., predicting user churn, forecasting sales, classifying item categories), projecting disjoint label spaces into a unified objective. This requires specialized decoders, such as text-encoded multi-task cross-attention modules or appending predictive tasks as “task tables,” [5] to unify all training signals into a standard masked token prediction problem. Uniform Representation of Sampled Subgraphs Because graph sampling is necessary for scale, the model must be robust to the variance introduced by sampling different temporal subgraphs. Standardizing the input spaces of these highly skewed relational graphs is necessary to ensure stable optimization. Balancing Loss and Gradient Scales Pretraining across diverse supervised tasks (regression vs. classification, high-frequency vs. low-frequency labels) creates conflicting gradient signals. A foundational architecture must modularize its learning objectives to balance gradient contributions and prevent dominant tasks from causing catastrophic interference with the generalized representation.
3
Modular Pretraining for RDL
To address the four pillars outlined above, we propose moving away from monolithic RDL architectures that attempt to solve feature mapping, temporal routing, and message passing simultaneously. Attempting to pass every cell of a database into a global graph transformer [33] results in an intractable O(N 2 ) memory bottleneck, while standard Heterogeneous GNNs fail to generalize across schemas. Instead, we adopt a divide-and-conquer formulation. Let a database be a col(k) mk (k) nk lection of tables D = {Tk }K k=1 with schemas Ck = {cj }j=1 and rows {ri }i=1 . We map D to a temporal relational graph G = (V, E, τ ) where each row corresponds to a node v = (k, i) ∈ V , PK–FK links induce edges (v → u) ∈ E, and τ : V → R assigns event times.
6
J. Peleška and G. Šír
Our modular pipeline then factorizes the predictor into three components: hv = fθ (rv ) ∈ Rd , {zv }v∈VS = gϕ GS , {hv }v∈VS , ŷu = dψ (zu ), (1) where fθ is a universal row encoder (schema-agnostic interface via fixed d), gϕ is a graph module applied to a sampled (temporal) subgraph GS , and dψ is a task head. This strict separation of concerns then decouples semantic feature extraction (Pillar 1) from structural and temporal propagation (Pillars 2–3), while enabling flexible multi-task optimization (Pillar 4). 3.1
The Universal Row Encoder and Statistical Contextualization
At the foundation of our modular pipeline is the Universal Row Encoder. Its objective is to resolve semantic and feature granularity by transforming any multi-typed database row into a fixed-size embedding h ∈ Rd . Let a row from a table with m columns be represented as m r = (vj , cj , t, sj , δj ) j=1 , (2) where vj is the raw cell value, cj is the column identifier (e.g., name), t is the table identifier (e.g., name), sj are precomputed column statistics, and δj ∈ {0, 1} indicates missingness. Specifically, the contents of the precomputed column statistics sj depend on the semantic type of the feature j: numerical columns utilize the mean, standard deviation, minimum, maximum, median, and the first and third quartiles; categorical and multi-categorical columns are summarized by their overall cardinality alongside precomputed text embeddings of their unique values; and timestamp columns capture temporal boundaries by explicitly storing the earliest and latest exact dates as well as the minimum and maximum years. For temporal databases, all statistics sj are computed in a split-aware manner (i.e., on the training portion, or only from past data for causal evaluation) to avoid leaking information from the future into the encoder. For each column j, we construct a contextualized column embedding xj ∈ Rd as a sum of type-specific value embeddings and metadata embeddings: xj = fstype(j) (vj ) + gname (cj ) + gtable (t) + gstats (sj ) + gmiss (δj ).
(3)
Here, fstype(j) denotes a semantic-type encoder. In our implementation, numerical values are standardized using sj and embedded by a small MLP; categorical values are mapped to trainable embeddings and combined with an embedding of the raw token; timestamps are embedded via a learned projection of normalized time (e.g., scaled to the [min, max] range in sj ) and a missingness flag. The functions gname and gtable create representation of schema text using a text summarization;2 gstats (sj ) and gmiss (δj ) are lightweight learned projections that inject distributional context and explicit missingness. 2
In the reported experiments we used pooled GloVe [23] embedding vectors, though more powerful models could be employed to further improve performance.
Universal Encoders for Modular Relational Deep Learning
Stats Linear Projection
Column Stats
h2 L
Schema Metadata
Text Embedder
Semantic Type
Type Embedding
. . . hn
Max Pool
Sum Pool
Attn. Pool
Row Embedding ∈ Rd
h1
Concat & Projection
Type-Specific Encoder
Raw Cell
Row Cells
Transformer Aggregator (Intra-Row Self-Attention)
Per-Column Statistical Contextualization
7
Fig. 3. Architecture of the Universal Row Encoder. Raw cell values are embedded and additively contextualized with distribution statistics, schema metadata, and semantic types. Contextualized column embeddings are aggregated via an intra-row transformer and a pooling mechanism to produce a fixed-size row representation.
The sequence (x1 , . . . , xm ) is then processed by an intra-row transformer to obtain contextualized token representations (z1 , . . . , zm ): (z1 , . . . , zm ) = Transformer(x1 , . . . , xm ).
(4)
Here, we use no positional encodings for columns. As a result, the encoder treats the row as a set of column tokens, making the transformation permutationequivariant with respect to column order3 and avoiding dependence on arbitrary schema ordering [34,18]. Finally, we derive a fixed-size row embedding by pooling over columns. Concretely, we use sum pooling, max pooling, and attentive pooling, psum =
m X j=1
zj ,
m
pmax = max zj , j=1
patt =
m X
αj zj , αj = softmax(q ⊤ zj ),
(5)
j=1
and project their concatenation to Rd : h = W [psum ∥ pmax ∥ patt ].
(6)
This design yields a uniform representation per row (independent of table width at the interface to downstream graph layers), while the intra-row attention cost depends only on the number of columns m in the processed table (not on the number of rows n or the global database size, avoiding the O(N 2 ) bottleneck). 3
While this design ensures transferability across diverse databases by preventing the model from using a specific table layout as a learning shortcut, it inevitably discards any latent information encoded in the original column order, such as implicitly ordered feature sequences or deliberately denormalized schemas.
8
3.2
J. Peleška and G. Šír
Graph-Agnostic Structural Modeling
After row encoding, each node v ∈ V is represented by hv ∈ Rd , yielding a unified feature space shared across tables. A downstream graph module gϕ then performs message passing on a sampled subgraph GS = (VS , ES ) to produce structural representations zv . To prevent temporal leakage, we enforce a causal constraint in sampling and propagation: for any prediction target node u at time τ (u), all nodes and edges used for its computation satisfy (v → u) ∈ ES ⇒ τ (v) ≤ τ (u), and more generally the sampled neighborhood must not include events from the future relative to the target time. This temporal constraint is applied when constructing GS (e.g., in a temporally filtered neighbor loader) before any GNN layers are evaluated. Note how the modular interface provides the architectural flexibility, where the same row embeddings can be used with homogeneous or heterogeneous GNN backbones. In a heterogeneous model, the message-passing parameters may depend on the edge types; in a homogeneous model, parameters are independent of the original schema. 3.3
Flexible Decoding
To address the challenges of unified optimization (Pillar 4), the modular framework extends to the decoder. Because the graph module outputs standardized node representations zu , the pipeline naturally supports multi-task learning with task-specific heads. Let {Tℓ }L ℓ=1 denote a set of supervised tasks, each defining a distribution over labeled nodes (u, y) ∼ Tℓ and a loss function Lℓ . We optimize min
θ,ϕ,{ψℓ }
L X
h i λℓ E(u,y)∼Tℓ Lℓ dψℓ (zu ), y ,
(7)
ℓ=1
where λℓ balances gradient contributions across tasks. 3.4
Supervised Pretraining Strategies
The outlined modularity then enables two distinct supervised pretraining regimes: Encoder-Only Pretraining In this regime, only the Universal Row Encoder is pretrained across diverse databases. It is then frozen and deployed as a generalpurpose neural feature extractor. This reduces the computational overhead of training RDL on new databases from scratch, while replacing manual feature engineering with robust, pretrained embeddings. End-to-End Pretraining In this regime, the Universal Row Encoder, the GNN architecture, and the task heads are pretrained simultaneously. This allows the row encoder to receive gradients shaped by structural and temporal motifs in the downstream graph module. In principle, such end-to-end pretraining can improve transfer to previously unseen relational structures, compared to pretraining the encoder in isolation.
Universal Encoders for Modular Relational Deep Learning Database
9
Domain #Tables #FK #Factual Diameter Cycle 1:1 1:N CTU Relational Databases
employee ergastf1 expenditures fnhk gosales grants lahman movielens restbase sakila sales sap seznam
Retail Sport Retail Medicine Retail Education Sport Entertain. Retail Retail Retail Retail Retail
6 13 3 3 4 12 25 7 3 16 4 4 4
6 19 2 2 3 11 31 6 3 23 3 3 3
16 82 19 21 17 30 319 14 10 75 12 35 10
3 3 2 2 2 4 6 4 1 7 2 2 2
✓ ✓ ✗ ✗ ✗ ✗ ✓ ✗ ✓ ✓ ✗ ✗ ✗
2 0 0 0 0 2 6 0 2 16 0 2 0
5 19 2 2 3 10 28 6 2 14 3 2 3
2 2 3 3 4
✗ ✓ ✓ ✓ ✓
0 0 0 2 4
2 11 13 11 13
RelBench Databases amazon avito f1 stack trial
Retail Retail Sport Education Medicine
3 8 9 7 15
2 11 13 12 15
10 23 45 33 110
Table 1. List of databases used in pretraining of the models.
4
Experiments
The experiments are designed to assess the framework’s modularity and crossdatabase transferability enabled by the Universal Row Encoder. We follow the standard predictive RDL setup, where each supervised task Tℓ is defined by (i) (ℓ) a set of target nodes Vtarget ⊆ V (e.g., rows of a specific table), (ii) a label (ℓ)
function y (ℓ) : Vtarget → Yℓ , and (iii) a prediction time tu = τ (u) (optionally with a fixed horizon) that determines the causal subgraph used for computing zu . For evaluation, we report a task-specific metric mℓ (ŷu , yu ) (ROC AUC for classification and MAE for regression in our setting) aggregated over held-out targets. The source code for all experiments can be accessed on GitHub4 . 4.1
Experimental Setup & Benchmarks
Our pretraining corpus consists of a diverse set of multi-tabular databases. We utilize the established RelBench [25] datasets alongside a suite of 13 temporal databases from the CTU Relational [19] suite accessed through the Redelex 4
https://github.com/jakubpeleska/redelex/tree/main/experiments/ universal_encoder
10
J. Peleška and G. Šír
framework [21]. All used databases are listed in Table 1 with their respective characteristics. For schema text (table/column names) and raw categorical tokens we use pooled GloVe vectors [23]. Structural neighborhood sampling is explicitly temporally constrained to prevent future information from entering message passing. In all settings we use batch size 128.
Pretraining The unified pretraining optimization is performed using the AdamW optimizer with a learning rate of 0.001 and weight decay of 0.1. To balance gradient scales across diverse pretraining tasks (Pillar 4), we employ a learning rate scheduler comprising a linear warmup phase starting at learning rate of 0.0001 for the first 10% of the maximum 50,000 training steps, followed by a Cosine Annealing decay. Models are pretrained on a set of binary classification and regression tasks, where the target values of the regression tasks are normalized for the purposes of the pretraining regime. Tasks are randomly permuted to mitigate a bias of the fixed order while ensuring uniform representation. To allow correct evaluation of foundational transferability, we employ a strict leave-one-database-out pretraining protocol. The models are pretrained on N −1 databases and subsequently evaluated on the held-out database. We are specifically using only the RelBench databases in place of the held-out database, meaning that the CTU Relational datasets are always used for the pretraining of the models.
Evaluation To assess downstream performance, we train the models on tasks defined on the database held out during pretraining. Similarly to the pretraining phase, we use the AdamW optimizer with a learning rate of 0.001 and a weight decay of 0.1, training for only 5000 steps to evaluate rapid adaptation. We report downstream performance using Receiver Operating Characteristic Area Under the Curve (ROC AUC) for binary classification and Mean Absolute Error (MAE), which are the standard evaluation metrics in RDL [25,16,32].
Models As the goal of this paper is not to train a single best model, but rather to demonstrate a modular approach to developing foundational models for RDL, we avoid extensive hyperparameter search. All models use fixed hyperparameters, with the only exception being the number of layers in the row encoder (1, 2, or 4). The Universal Row Encoder’s transformer module utilizes 8 attention heads with a dropout rate of 0.1. All models employ a GraphSAGE [8] convolution backbone with 2 message-passing layers and an inner embedding dimension of 512. The task-specific baseline mirrors the prior model architecture [25] often used as the RDL baseline [4,21], a heterogeneous GNN that employs a dedicated row encoder for each table, with learnable weights for each column based on its semantic type. Lastly, for all tasks, we employ a single fully connected layer to map the model’s latent space to the prediction values, intentionally keeping the prediction head capacity small.
Universal Encoders for Modular Relational Deep Learning
11
Fig. 4. Pretraining loss trajectories across multiple hold-out scenarios. The plots compare the training and validation loss of the Universal Row Encoder coupled with a Shared Homogeneous GNN and a Task-specific Heterogeneous GNN. Both architectural variants demonstrate stable convergence across different hold-out scenarios.
4.2
Pretraining Universal Encoders
We compare two message-passing backbones attached to the Universal Row Encoder: a simplified Homogeneous GNN, with shared weights across node and edge types, and a task-specific Heterogeneous GNN with relation-dependent parameters. Figure 4 shows that both variants converge stably across leave-one-databaseout pretraining runs. The heterogeneous backbone achieves slightly lower loss trajectories, consistent with its higher relational capacity, while the homogeneous backbone remains competitive and notably stable across hold-out databases. We additionally include a rel-all setting (training only on CTU Relational while holding out all RelBench datasets) to contextualize loss scales under maximal domain shift. 4.3
Transferability with Frozen Encoders
To evaluate transferability, we assess performance on databases held out from pretraining under three regimes: (1) head-only transfer, where the encoder and homogeneous GNN are frozen and only a task head dψℓ is trained; (2) frozen encoder, where fθ is fixed but the downstream GNN and head are trained; and
12
J. Peleška and G. Šír
Fig. 5. Downstream task evaluation of the Universal Row Encoder in different training regimes. The plots show the evaluation metrics (ROC AUC for classification, MAE for regression) on unseen databases across evaluation steps.
(3) frozen encoder with task-specific pretraining, where the encoder is fixed but was pretrained jointly with task-specific heterogeneous GNNs. We compare these to an end-to-end task-specific HGNN trained from scratch on the target database (after 1000 and 5000 steps). Figure 5 shows that head-only transfer is the most challenging regime, yet it can still provide non-trivial signal, particularly on some regression tasks. In contrast, regimes (2) and (3) exhibit consistent few-shot behavior on classification tasks and can match or surpass the task-specific baseline on selected tasks (e.g., user-visits, post-votes). Overall, these results suggest that a universal row encoder can reduce the burden of schema-specific input layers and improve rapid adaptation on unseen databases. 4.4
Finetuning the Pretrained Models
We compare keeping the pretrained row encoder frozen versus finetuning it jointly with the downstream GNN and task head. Figure 6 shows only minor differences between the two regimes across the evaluated tasks and backbones. This suggests that most of the transferable benefit is already captured during pretraining, and that downstream training can often focus on adapting the structural module and task head. From a practical perspective, freezing fθ simplifies deployment and reduces the computational cost of adapting the model to a new database.
Universal Encoders for Modular Relational Deep Learning
13
Fig. 6. Comparison of frozen versus finetuned Universal Row Encoder regimes on downstream tasks. Finetuning the pretrained encoder alongside the GNN backbone yields minor to none additional performance relative to the completely frozen row encoder weights.
4.5
Memory-Performance Tradeoff
We evaluate the memory efficiency of our universal approach compared to specialized task-specific baselines. Table 2 reports the predictive performance (MAE) alongside the model footprint (in MB) across various regression tasks. As expected, replacing a highly parameterized, schema-specific architecture with a single frozen universal encoder involves a practical trade-off: the results demonstrate a significant memory efficiency gain at the cost of a mild drop in MAE accuracy. Across the reported regression tasks, the Specialized model consistently achieves lower error rates. For instance, on the rel-trial dataset, the Universal model exhibits a roughly 18% relative increase in MAE compared to the Specialized baseline. However, this accuracy cost comes with a notable reduction in model size. On rel-trial, the Universal model requires only 34.06 MB compared to 368.81 MB for the Specialized model—a nearly 10× reduction. Such reductions are important when deploying models across many databases, where per-schema encoders quickly become computationally intractable. The reduction follows from parameter sharing: specialized baselines scale PK (k) roughly with one row encoder per table, |Θspec | ≈ k=1 |θrow | + |ϕ|, whereas the universal approach reuses a single encoder, |Θuni | ≈ |θ| + |ϕ|.
14
J. Peleška and G. Šír
Dataset
Task
rel-amazon item-ltv user-ltv rel-avito
ad-ctr
rel-f1
driver-position
rel-stack
post-votes
rel-trial
site-success study-adverse
Split
Specialized MAE Size MB
MAE
Universal Size MB
val test val test val test val test val test val test val test
51.44 56.35 13.19 15.55 0.03 0.04 3.20 3.62 0.06 0.06 0.36 0.39 48.27 47.04
56.89 61.53 14.15 16.79 0.04 0.04 3.29 4.05 0.06 0.07 0.45 0.47 54.37 55.79
18.28 18.28 18.28 18.28 34.06 34.06 23.54 23.54 18.28 18.28 34.06 34.06 34.06 34.06
51.58 51.58 51.58 51.58 121.83 121.83 179.03 179.03 135.19 135.19 368.81 368.81 368.81 368.81
Table 2. Comparison of the of the Specialized model—schema-specific row encoder with HGNN—vs. the Universal model—Universal Row Encoder with Homogeneous GNN and frozen weigh—in terms of the best results on the regression downstream tasks and memory footprint (model size in MB). The Universal approach yields significant memory efficiency gains (up to 10× smaller footprint) at the cost of a mild drop in predictive accuracy.
5
Related Work
The RDL paradigm was formalized [35,5] to connect deep learning and relational databases. The field is supported by benchmarks [28,7,25] as well as practical frameworks for building temporal relational graphs [21]. A large body of work explores architectures for learning on relational graphs, ranging from schema-specific relational GNNs [2] to approaches that reuse strong tabular learners [17,22]. More recently, graph transformers and LLM-inspired models have been adapted to relational settings [4,32,16]. In parallel, the community is slowly moving toward foundational RDL models trained across many databases [31,30,4,20]. These efforts often rely on monolithic transformer-based designs that jointly model cell-level semantics and relational propagation. Our work is complementary: we keep the relational message-passing module flexible, but focus on a universal, distribution-aware row encoder as a reusable backend. Synthetic relational data generation [12,14] and further formalization of predictive RDL tasks [13] are likely to be important enablers for scaling such pretraining regimes.
Universal Encoders for Modular Relational Deep Learning
6
15
Conclusion
In this work, we identified generalization bottlenecks in Relational Deep Learning (RDL) and distilled four pillars for relational foundation models: semantic granularity, structural topology, temporal causality, and unified optimization. To address these, we proposed a modular RDL architecture that decouples rowlevel semantic encoding from graph-level message passing. We then introduced a new baseline implementation of the approach, with a Universal Row Encoder at its core. The encoder combines type-specific value embeddings with schema text and column-level distribution statistics to produce a fixed-size row representation that can be reused across different databases. Empirically, we showed that pretrained row encoding can accelerate adaptation on held-out databases and significantly reduce model sizes relative to schema-specific baselines, at the cost of but a mild drop in predictive accuracy. Limitations. Our four-pillar framework establishes a conceptual roadmap for RDL foundation models; however, our current technical implementation primarily addresses the first pillar (semantic granularity) and the schema-agnostic requirement of the second pillar. The remaining aspects of structural topology and temporal causality are currently handled via data representation mechanisms—such as explicit primary-foreign key graphs and temporally constrained sampling—rather than being intrinsically embedded within the model’s architecture. Furthermore, regarding the fourth pillar (unified optimization), our approach was limited to target value normalization across just two task types: binary classification and regression. We did not explore loss balancing for broader task categories, such as multi-label classification or link prediction, nor did we implement a unified prediction head. Nevertheless, formalizing these pillars provides a necessary, unified vocabulary to guide the resolution of these bottlenecks in future research beyond Universal Row Encoder. Future Directions. While our baseline demonstrates the viability of a modular approach, several immediate avenues for refinement remain. First, future work should formally ablate the contextualization components in Equation 3—namely the global statistics, schema names, and missingness indicators—alongside the intra-row pooling strategies (Equation 6) to rigorously isolate their individual contributions. Second, while precomputed summary statistics serve as an effective heuristic, exploring more principled mechanisms such as learned quantile features or percentile positions will be vital for capturing complex, non-parametric distributions. Similarly, replacing lightweight GloVe vectors with modern sentence encoders (e.g., MiniLM [29], E5 [27]) promises straightforward performance gains. Looking beyond the encoder, our framework opens broader research directions. To fully realize the desiderata defined by the four pillars, subsequent work must address the challenge of scaling model capacity [4], potentially by leverag-
16
J. Peleška and G. Šír
ing synthetic relational data [14,9] to satisfy the requirements for substantially larger, multi-domain training corpora [26]. Acknowledgments. This work has received funding from the Czech Science Foundation grant No. 26-22501S. Computational infrastructure was provided by the OP VVV funded project CZ.02.1.01/0.0/0.0/16_019/0000765 “Research Center for Informatics”. Ethical Considerations. As foundational Relational Deep Learning (RDL) models scale by training on massive, multi-domain databases, they inherently risk capturing and propagating sensitive information. Real-world relational databases frequently contain Personally Identifiable Information (PII) and protected attributes across highstakes domains like healthcare, finance, and criminal justice. The deployment of universal row encoders introduces the risk of unintended data memorization. Furthermore, models explicitly relying on global column statistics may inadvertently encode and amplify structural biases present in historical data distributions. Future research scaling these models must prioritize privacy-preserving pretraining objectives and develop robust auditing tools to ensure foundational RDL models are deployed equitably and securely. Generative AI Statement. Generative AI tools were used to assist in preparing this paper. They were employed responsibly to uphold the integrity of the submission, specifically restricted to enhancing the readability of the text.
References 1. Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D.M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., Amodei, D.: Language models are few-shot learners. Advances in Neural Information Processing Systems (2020) 2. Chen, T., Kanatsoulis, C., Leskovec, J.: RelGNN: Composite Message Passing for Relational Deep Learning (Feb 2025). https://doi.org/10.48550/arXiv.2502. 06784, arXiv:2502.06784 [cs] 3. Cvitkovic, M.: Supervised learning on relational databases with graph neural networks. arXiv preprint arXiv:2002.02046 (2020) 4. Dwivedi, V.P., Jaladi, S., Shen, Y., López, F., Kanatsoulis, C.I., Puri, R., Fey, M., Leskovec, J.: Relational Graph Transformer (May 2025). https://doi.org/ 10.48550/arXiv.2505.10960, arXiv:2505.10960 [cs] 5. Fey, M., Hu, W., Huang, K., Lenssen, J.E., Ranjan, R., Robinson, J., Ying, R., You, J., Leskovec, J.: Position: Relational Deep Learning - Graph Representation Learning on Relational Databases. In: Proceedings of the 41st International Conference on Machine Learning. pp. 13592–13607. PMLR (Jul 2024), https://proceedings.mlr.press/v235/fey24a.html, iSSN: 2640-3498 6. Gama, J., Žliobaitē, I., Bifet, A., Pechenizkiy, M., Bouchachia, A.: A survey on concept drift adaptation. ACM Computing Surveys 46(4), 44:1–44:37 (2014)
Universal Encoders for Modular Relational Deep Learning
17
7. Gu, J., Ranjan, R., Kanatsoulis, C., Tang, H., Jurkovic, M., Hudovernik, V., Znidar, M., Chaturvedi, P., Shroff, P., Li, F., Leskovec, J.: RelBench v2: A Large-Scale Benchmark and Repository for Relational Data (Feb 2026). https: //doi.org/10.48550/arXiv.2602.12606, arXiv:2602.12606 [cs] 8. Hamilton, W.L., Ying, R., Leskovec, J.: Inductive Representation Learning on Large Graphs. In: Advances in Neural Information Processing Systems (2017) 9. Hollmann, N., Müller, S., Purucker, L., Krishnakumar, A., Körfer, M., Hoo, S.B., Schirrmeister, R.T., Hutter, F.: Accurate predictions on small data with a tabular foundation model. Nature 637(8045), 319–326 (Jan 2025). https: //doi.org/10.1038/s41586-024-08328-6, https://www.nature.com/articles/ s41586-024-08328-6, publisher: Nature Publishing Group 10. Hu, Z., Dong, Y., Wang, K., Sun, Y.: Heterogeneous graph transformer. In: Proceedings of the web conference 2020. pp. 2704–2710 (2020) 11. Huang, X., Khetan, A., Cvitkovic, M., Karnin, Z.: Tabtransformer: Tabular data modeling using contextual embeddings. In: arXiv preprint arXiv:2012.06678 (2020) 12. Hudovernik, V.: Deep Learning Methods for Synthetic Relational Data Generation. thesis, Univerza v Ljubljani, Fakulteta za računalništvo in informatiko (2025) 13. Kocijan, V., Sunil, J., Lenssen, J.E., Deb, V., Xe, X., Gomez, F.R., Fey, M., Leskovec, J.: Predictive Query Language: A Domain-Specific Language for Predictive Modeling on Relational Databases (Feb 2026). https://doi.org/10.48550/ arXiv.2602.09572, arXiv:2602.09572 [cs] 14. Kothapalli, V., Ranjan, R., Hudovernik, V., Dwivedi, V.P., Hoffart, J., Guestrin, C., Leskovec, J.: PluRel: Synthetic Data unlocks Scaling Laws for Relational Foundation Models (Feb 2026). https://doi.org/10.48550/arXiv.2602.04029, arXiv:2602.04029 [cs] 15. Kramer, S., Lavrač, N., Flach, P.: Propositionalization approaches to relational data mining. Relational data mining pp. 262–291 (2001) 16. Lachi, D., Mohammadi, M., Meyer, J., Arora, V., Palczewski, T., Dyer, E.L.: Integrating Temporal and Structural Context in Graph Transformers for Relational Deep Learning (Nov 2025). https://doi.org/10.48550/arXiv.2511.04557, http://arxiv.org/abs/2511.04557, arXiv:2511.04557 [cs.LG] 17. Lachi, V., Longa, A., Bevilacqua, B., Lepri, B., Passerini, A., Ribeiro, B.: Boosting Relational Deep Learning with Pretrained Tabular Models (Apr 2025). https: //doi.org/10.48550/arXiv.2504.04934, arXiv:2504.04934 [cs] 18. Lee, J., Lee, Y., Kim, J., Kosiorek, A., Choi, S., Teh, Y.W.: Set transformer: A framework for attention-based permutation-invariant neural networks. In: Proceedings of the 36th International Conference on Machine Learning (2019) 19. Motl, J., Schulte, O.: The CTU Prague Relational Learning Repository. arXiv preprint arXiv:1511.03086 (2015) 20. Peleška, J., Šír, G.: Task-agnostic contrastive pretraining for relational deep learning. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases. pp. 623–638. Springer (2025) 21. Peleška, J., Šír, G.: ReDeLEx: A Framework for Relational Deep Learning Exploration. In: Ribeiro, R.P., Pfahringer, B., Japkowicz, N., Larrañaga, P., Jorge, A.M., Soares, C., Abreu, P.H., Gama, J. (eds.) Machine Learning and Knowledge Discovery in Databases. Research Track. pp. 438–456. Springer Nature Switzerland, Cham (Sep 2025) 22. Peleška, J., Šír, G.: Tabular Transformers Meet Relational Databases. ACM Trans. Intell. Syst. Technol. 16(5), 115:1–115:24 (Sep 2025). https://doi.org/10.1145/ 3749991
18
J. Peleška and G. Šír
23. Pennington, J., Socher, R., Manning, C.D.: GloVe: Global Vectors for Word Representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) (2014) 24. Ranjan, R., Hudovernik, V., Znidar, M., Kanatsoulis, C., Upendra, R., Mohammadi, M., Meyer, J., Palczewski, T., Guestrin, C., Leskovec, J.: Relational Transformer: Toward Zero-Shot Foundation Models for Relational Data (Oct 2025). https://doi.org/10.48550/arXiv.2510.06377, arXiv:2510.06377 [cs] 25. Robinson, J., Ranjan, R., Hu, W., Huang, K., Han, J., Dobles, A., Fey, M., Lenssen, J.E., Yuan, Y., Zhang, Z., He, X., Leskovec, J.: RelBench: A Benchmark for Deep Learning on Relational Databases. In: The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2024) 26. Vogel, L., Bodensohn, J.M., Binnig, C.: WikiDBs: A Large-Scale Corpus Of Relational Databases From Wikidata. In: Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J., Zhang, C. (eds.) Advances in Neural Information Processing Systems. vol. 37, pp. 41186–41201. Curran Associates, Inc. (2024), https://proceedings.neurips.cc/paper_files/paper/2024/file/ 4847447cae911dde321ca19afe7d8707-Paper-Datasets_and_Benchmarks_Track. pdf 27. Wang, L., Yang, N., Huang, X., Yang, L., Majumder, R., Wei, F.: Multilingual e5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672 (2024) 28. Wang, M., Gan, Q., Wipf, D., Cai, Z., Li, N., Tang, J., Zhang, Y., Zhang, Z., Mao, Z., Song, Y., et al.: 4dbinfer: A 4d benchmarking toolbox for graph-centric predictive modeling on rdbs. Advances in Neural Information Processing Systems 37, 27236–27273 (2024) 29. Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., Zhou, M.: Minilm: Deep selfattention distillation for task-agnostic compression of pre-trained transformers. Advances in neural information processing systems 33, 5776–5788 (2020) 30. Wang, Y., Wang, X., Gan, Q., Wang, M., Yang, Q., Wipf, D., Zhang, M.: Griffin: Towards a Graph-Centric Relational Database Foundation Model (May 2025). https://doi.org/10.48550/arXiv.2505.05568, arXiv:2505.05568 [cs] 31. Wehrstein, J., Binnig, C., Özcan, F., Vasudevan, S., Gan, Y., Wang, Y.: Towards foundation database models. In: CIDR (2025) 32. Wu, F., Dwivedi, V.P., Leskovec, J.: Large language models are good relational learners. In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 7835–7854 (2025) 33. Yun, S., Jeong, M., Kim, R., Kang, J., Kim, H.J.: Graph Transformer Networks. In: Advances in Neural Information Processing Systems. vol. 32. Curran Associates, Inc. (2019), https://proceedings.neurips.cc/paper/2019/hash/ 9d63484abb477c97640154d40595a3bb-Abstract.html 34. Zaheer, M., Kottur, S., Ravanbakhsh, S., Póczos, B., Salakhutdinov, R.R., Smola, A.J.: Deep sets. In: Advances in Neural Information Processing Systems (2017) 35. Zahradník, L., Neumann, J., Šír, G.: A Deep Learning Blueprint for Relational Databases. In: NeurIPS 2023 Second Table Representation Learning Workshop (2023) 36. Zhang, C., Song, D., Huang, C., Swami, A., Chawla, N.V.: Heterogeneous Graph Neural Network. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. pp. 793–803. KDD ’19, Association for Computing Machinery, New York, NY, USA (Jul 2019). https: //doi.org/10.1145/3292500.3330961