Towards Foundation Models for Relational Databases with Language Models and Graph Neural Networks Jingcheng Wu1 , Ratan Bahadur Thapa1 , Mojtaba Nayyeri1 , Lucas Etteldorf1 , Max Finkenbeiner1 , Fabian Leeske1 and Steffen Staab1,2 1 2
University of Stuttgart, Stuttgart, Germany Web and Internet Science Research Group, University of Southampton, Southampton, United Kingdom
Abstract
Relational databases store much of the world’s structured information, and they are essential for driving complex predictive applications. However, deep learning progress on relational data remains limited, as conventional approaches flatten databases into single tables via manual feature engineering, discarding relational context. Relational deep learning (RDL) addresses this by modeling databases as relational entity graphs (REGs) for graph neural networks (GNNs), but remains task- and database-specific. To combine the strengths of both paradigms, we propose a hybrid architecture combining a fine-tuned BART encoder to capture intra-row semantics with a GraphSAGE-based GNN over REGs to inject relational context. Experiments on RelBench show that the GNN substantially enriches BART’s row embeddings, achieving a ROC-AUC of 67.40 on the driver-dnf task from the rel-f1 dataset. This performance is competitive with supervised baselines such as LightGBM (68.86) and narrows the gap to RDL (72.62) to within 5.22 points, though a substantial gap remains to state-of-the-art foundation models such as KumoRFM (82.63). These results suggest that lightweight hybrid LM-GNN architectures offer a promising and resource-efficient path towards foundation models for relational databases.
Keywords
Foundation Models, Relational Databases, Relational Deep Learning, Language Models, Graph Neural Networks
1. Introduction Relational databases store much of the world’s structured data in multiple tables, with rows in different tables connected using primary and foreign keys [1, 2]. They underpin large-scale information systems in domains such as e-commerce, banking, and healthcare. Many real-world predictive tasks depend on such relational data, including forecasting future product sales, predicting user churn, or estimating the risk of discharging a patient. To perform such tasks effectively, it is essential to leverage both the data in the tables and the relationships encoded in the database schema. Although the success of deep learning methods on unstructured data makes its application to relational databases highly desirable, these methods have inherently struggled with the complexity of relational structures [1]. Since existing tabular models cannot directly learn across multiple interconnected tables, the standard practice relies on manual feature engineering. Relational databases are typically flattened into a single table through joins and aggregations before applying conventional machine learning methods. This flattening procedure requires domain expertise, is slow and error-prone, and severely degrades the relational context provided by primary-foreign key links. To address these limitations, Fey et al. [1] introduced relational deep learning (RDL), which models databases as relational entity graphs (REGs) and applies graph neural networks (GNNs). Although RDL improves upon feature engineering across most RelBench tasks, its models are trained separately per database and task, and do not learn transferable representations. While this limitation has recently
Preprint of a paper accepted at the 1st Workshop on Extraction from Triplet Text-Table-Knowledge Graph and associated Challenge (TRIPLET), co-located with ESWC 2026, May 10–14, 2026, Dubrovnik, Croatia $ [email protected] (J. Wu); [email protected] (R. B. Thapa); [email protected] (M. Nayyeri); [email protected] (L. Etteldorf); [email protected] (M. Finkenbeiner); [email protected] (F. Leeske); [email protected] (S. Staab) © 2026 Copyright for this paper by its authors. This author version is made available under the Creative Commons Attribution 4.0 International License (CC BY 4.0).
motivated the first relational foundation models [3, 4], these architectures are highly complex and computationally intensive, posing significant barriers to widespread adoption. Seeking a more lightweight and accessible alternative, we draw inspiration from the hybrid LM-GNN concept initially proposed by Vogel et al. [5]. As their framework focused exclusively on generationbased data preparation tasks, such as missing value imputation, it cannot be directly applied to generalpurpose predictive transfer. To bridge this gap, we propose an extended architecture combining two core components: a fine-tuned BART encoder to extract row-level semantics, and a GraphSAGE-based GNN over REGs to inject relational context. To assess whether this hybrid approach can serve as a foundation towards general-purpose relational foundation models, we evaluate our architecture on the driver-dnf node classification task from the rel-f1 dataset in RelBench. Our main contributions are as follows: 1. We propose a novel LM-GNN hybrid architecture that seamlessly integrates a pre-trained language model (BART) with a GraphSAGE-based GNN. The language model captures deep intra-row semantics, while the GNN enriches these embeddings with structural relational context. Coupled with self-supervised objectives, namely schema-aware token masking for the LM and masked feature reconstruction for the GNN, this framework generates contextualized representations easily adaptable to downstream tasks with minimal computational overhead. 2. We provide a systematic empirical evaluation of our architecture on the RelBench benchmark. Our model achieves a highly competitive ROC-AUC score of 67.40 on the driver-dnf task from the rel-f1 dataset. Furthermore, our extensive ablation study demonstrates that explicitly incorporating relational context via message passing yields substantial ROC-AUC improvements compared to relying solely on isolated BART or random embeddings, highlighting the critical necessity of structural message passing. 3. We outline a strategic roadmap for future relational foundation models based on an in-depth analysis of our framework’s strengths and limitations. Specifically, we identify key research directions, including the joint training of the LM-GNN pipeline and architectural adaptations required to scale across diverse relational databases.
2. Background We adopt the formalism of Fey et al. [1]. A relational database (𝒯 , ℒ) consists of tables 𝒯 = {𝑇1 , . . . , 𝑇𝑛 } and links ℒ ⊆ 𝒯 ×𝒯 , where a link (𝑇fkey , 𝑇pkey ) ∈ ℒ exists when a foreign key column of 𝑇fkey references the primary key of 𝑇pkey . Each table 𝑇 = {𝑣1 , . . . , 𝑣𝑛𝑇 } contains entities (rows), each with a primary key 𝑝𝑣 , foreign keys 𝐾𝑣 , attributes 𝑥𝑣 = {𝑥1𝑣 , . . . , 𝑥𝑑𝑣𝑇 }, and an optional timestamp 𝑡𝑣 . All entities in the same table share the same columns, though individual values may differ or be missing. Schema Graph and Relational Entity Graph. The schema graph (𝒯 , ℛ) captures the table-level structure, with tables as nodes and bidirectional edges ℛ = ℒ ∪ ℒ−1 (where ℒ−1 contains the inverse of each link). The relational entity⋃︀ graph (REG) is a heterogeneous graph 𝐺 = (𝒱, ℰ, 𝜑, 𝜓) suitable for GNN processing. Its node set 𝒱 = 𝑇 ∈𝒯 𝑇 contains all database rows, and its edge set ℰ connects entity pairs linked by primary-foreign key relationships. Type mapping functions 𝜑 : 𝒱 → 𝒯 and 𝜓 : ℰ → ℛ assign each node and edge to its corresponding schema graph element, enabling type-specific handling by GNNs. In our architecture, we use node-type-specific linear layers (Section 4.1). Each node 𝑣 ∈ 𝒱 carries an embedding ℎ𝑣 ∈ R𝑑 . Initial embeddings are obtained from our fine-tuned BART encoder (Section 4.3.1) and enriched via the GNN (Section 4.3.2).
3. Related Work Foundation Models and Tabular Learning. Foundation models are large models pre-trained on broad data with self-supervised objectives and adaptable to diverse downstream tasks via fine-tuning [6].
They have profoundly transformed NLP and vision. GPT-3 demonstrated that scaling enables in-context learning and few-shot generalization [7], while models such as CLIP and its variants [8, 9] extend these principles to multimodal settings. Their effectiveness stems from self-supervised representation learning [10] combined with transfer learning across datasets and tasks [11]. Despite these successes in unstructured domains, deep learning has long underperformed tree-based methods such as XGBoost [12] due to challenges including small dataset sizes, absence of spatial or sequential structure, and heterogeneous feature types mixing numerical and categorical attributes [13, 14]. Recent transformer-based methods, including TabTransformer [15], FT-Transformer [16], and SAINT [17], have narrowed this gap via attention mechanisms that capture complex feature interactions. More recently, TabPFN [18] emerged as a tabular foundation model that outperforms prior methods on datasets up to 10,000 samples. However, all these advances remain limited to single tables and cannot exploit the relational structure of multi-table databases. Language Models for Structured Data. A parallel line of work adapts pre-trained language models to structured inputs by linearizing rows and schema information. At the core of many such approaches are foundational models like BART [19]. As a sequence-to-sequence model pre-trained as a denoising autoencoder, BART produces rich contextual embeddings by reconstructing corrupted input sequences. For single tables, TaBERT [20] jointly models tabular inputs and textual inputs, while TURL [21] learns contextualized table representations. Extending to relational databases, RPT [22] applies sequenceto-sequence models to data preparation tasks such as missing value imputation. While these works demonstrate the feasibility of language models for tabular data, they remain fundamentally limited to isolated tables or data preparation procedures rather than general-purpose predictive transfer. Graph Neural Networks and Relational Deep Learning. GNNs extend deep learning to graphstructured data, where message-passing architectures iteratively update node embeddings by aggregating neighborhood information [2, 23, 24]. GraphSAGE [25] introduced inductive learning via neighborhood sampling, enabling generalization to unseen nodes. This property is crucial for the development of foundation models. Self-supervised objectives such as GraphMAE [26], a masked graph autoencoder, further improve generalization and inspired our pre-training procedure. RDL [1] applies these ideas to relational databases by representing them as temporal, heterogeneous REGs and training GNNs end-to-end. It automates the construction of training labels from the database itself, encodes rows into initial node features, applies message-passing GNNs, and trains task-specific prediction heads. While RDL outperforms manual feature engineering with reduced development time, it remains database-specific and task-specific. These models are trained separately for each setting and lack the capacity to transfer across domains [2]. Foundation Models for Relational Databases. Building on RDL, recent efforts mark the first steps towards relational foundation models. Griffin [3] follows a graph-centric approach utilizing unified input encoders for diverse data types, alongside standardized task decoders. Its advanced architecture incorporates cross-attention for node feature aggregation alongside relation-aware message passing. After pre-training on diverse single-table datasets and joint supervised fine-tuning on selected RDB subsets (the Griffin-RDB-SFT variant in [3]), followed by task-specific fine-tuning, Griffin achieves an average ROC-AUC of 75.00 on classification tasks. For the zero-shot relational baseline in Section 5.2, we report the Griffin-pretrained variant (pre-trained exclusively on single-table datasets, as defined in [3]). KumoRFM [4], a proprietary commercial model, encodes database rows via a shared encoder tailored to semantic column types. It then applies a relational graph transformer to the resulting graph representation. This design enables in-context learning, allowing KumoRFM to generalize to previously unseen databases and tasks. Without prior training on test datasets, it achieves an average classification ROC-AUC of 76.71 and reaches 81.14 after fine-tuning.
Our work builds on the earlier vision of Vogel et al. [5], who proposed combining an LM encoder for row embeddings with a GNN for relational context. Their prototype demonstrated promise on data engineering tasks such as missing value imputation but did not address predictive tasks or broader generalization. We extend this vision by evaluating the hybrid architecture in a predictive setting on RelBench, positioning it as a lightweight, accessible alternative to the complex architectures of Griffin and KumoRFM.
4. Methodology We extend the hybrid LM-GNN architecture of Vogel et al. [5] by integrating RDL principles [1] and introducing self-supervised pre-training objectives. We represent relational databases as REGs, with initial node features obtained from a fine-tuned BART encoder [19] and subsequently enriched via message passing within a heterogeneous GNN. This two-stage pipeline first captures row-level semantics via BART fine-tuning, and then injects relational dependencies through the GNN’s message passing, optimized via self-supervised pre-training.
4.1. Architecture As illustrated in Fig. 1, our framework consists of two components: (1) a fine-tuned BART encoder that linearizes database rows into schema-aware strings and produces row-level embeddings capturing intrarow semantics, and (2) a GraphSAGE-based GNN operating on REGs that enriches these embeddings with relational context via message passing.
HeteroGraphSAGE
Transactions userID
date
#11
#01
20/08/2025
#12
#01
21/08/2025
…
…
…
Users userID
name
address
#01
Alice
Stuttgart
#02
Bob
Munich
…
…
…
row-wise serialization
Pretrained BART
transactionID
Heterogeneous Graph
Node Type: Transactions
Node Type specific Linear Layer (256)
Node Type specific Linear Layer (256)
Shared SAGEConv Layer1
Shared SAGEConv Layer 2
Node Type: Users
Linear Decoder (1024)
Figure 1: Overview of the hybrid architecture. A fine-tuned BART encoder generates row-level embeddings from linearized database rows, which serve as initial node features in the relational entity graph (REG). Nodetype-specific linear layers project the 1024-dimensional BART embeddings to the 256-dimensional hidden space. Two shared SAGEConv layers then perform message passing across all edge types, and a linear decoder maps the enriched embeddings back to 1024 dimensions for reconstruction loss computation.
We choose BART for its denoising pre-training that yields rich contextual embeddings and its demonstrated effectiveness on relational tasks [5]. For the GNN, we adopt GraphSAGE [25] for its inductive generalization to unseen nodes, scalable mini-batch training, and strong benchmark performance [27]. To handle heterogeneous graph data, we employ a HeteroGraphSAGE model via PyTorch Geometric
[28] with node-type-specific linear layers and shared SAGEConv layers across all edge types to improve generalization to unseen schemas. The GNN output dimensionality is set to 1024, matching the BART embeddings for reconstruction loss computation.
4.2. Data Preparation We use the RelBench benchmark [29], selecting 6 of its 7 databases (rel-trial, rel-avito, rel-hm, rel-amazon, rel-event, rel-stack) for pre-training and reserving rel-f1 for downstream evaluation to ensure an unbiased generalization test. Preprocessing for BART. We randomly sample 100,000 rows evenly distributed across the 6 databases and their tables. Each row is linearized into a schema-aware sequence: <table> Table Name <attr> Attr1 Name <value> Attr1 Value . . . The special tokens <table>, <attr>, and <value> are added to BART’s tokenizer vocabulary as new tokens, and the model’s token embeddings are resized accordingly. Missing and NaN values are replaced with empty strings. This format encodes table names, attribute names, and values, enabling BART to capture schema-data dependencies. The resulting rows are split into a 70/10/20 ratio for training, validation, and testing sets. Preprocessing for the GNN. Each database is transformed into a heterogeneous REG, with tables as distinct node types and primary-foreign key relationships as edges. Rows are encoded using the fine-tuned BART model, yielding 1024-dimensional node embeddings. For computational feasibility, larger tables are subsampled to limit graph size, producing graphs of approximately 4 to 6 million nodes each and totaling around 28 million nodes and 40 million edges across all six databases (Table 1). Table 1 Statistics of the six RelBench databases used for GNN pre-training. Node and edge counts reflect the truncated versions used in our experiments (see Section 4.2). Dataset
Nodes
Edges
rel-amazon rel-avito rel-event rel-hm rel-trial rel-stack
4,589,766 5,142,166 6,001,744 4,047,698 4,921,015 4,023,229
7,000,000 9,488,416 6,020,839 7,000,000 4,671,285 5,870,601
Total
28,725,618
40,051,141
4.3. Training Pipeline 4.3.1. Stage 1: Fine-tuning BART We fine-tune a pre-trained BART model1 [19] on the linearized rows, which are tokenized and truncated to a maximum sequence length of 1024 tokens. Instead of standard token-level masking, we apply a schema-aware masking strategy at the level of semantic units. Table names are masked with probability 0.30, attribute names with 0.20, and cell values with 0.40. At least one element is masked per row, and empty values are skipped. The model is trained to reconstruct the original sequence using cross-entropy loss. Fine-tuning proceeds for 50 epochs with the AdamW optimization, a cosine learning rate schedule with warmup, and an effective batch size of 128. Masking is applied dynamically per epoch on the 1
We use the facebook/bart-large checkpoint from Hugging Face.
training set. Validation and test sets use static masks. The resulting encoder produces 1024-dimensional row embeddings via mean-pooling of the last hidden states of the tokens, which serve as initial node features for the GNN stage. 4.3.2. Stage 2: Training the GNN The GNN is trained with a masked value reconstruction objective. For each node, random feature dimensions are masked and the loss is computed only on these dimensions, forcing the model to infer missing information via message passing from neighbors. Unlike standard reconstruction objectives that consider the entire embedding vector, restricting the loss to masked dimensions prevents the model from trivially copying observed values and explicitly encourages it to leverage relational context. We adopt a weighted combination of scaled cosine error [26] and MSE: (1)
ℒ = 𝛼 ℒcos + (1 − 𝛼) ℒmse , ^𝑖: where, for masked dimensions ℳ of each node 𝑖 with true features x𝑖 and reconstruction x (︂ )︂ 𝑁 𝑁 𝛾 ^ ℳ · xℳ 1 ∑︁ x 1 ∑︁ ℳ 2 ℒcos = 1 − ℳ 𝑖 ℳ𝑖 , ℒmse = ‖^ x𝑖 − x ℳ 𝑖 ‖2 . 𝑁 𝑁 ‖^ x𝑖 ‖ ‖x𝑖 ‖ + 𝜖 𝑖=1
(2)
𝑖=1
Here 𝜖 = 10−6 ensures numerical stability, 𝛾 = 2 emphasizes larger cosine errors following Hou et al. [26], and 𝑁 is the batch size. Training utilizes the NeighborLoader utility from PyTorch Geometric [28] with a neighborhood sampling strategy of (20, 10) (up to 20 first-hop and 10 second-hop neighbors per node) with mean aggregation and self-loops. To prevent catastrophic forgetting across databases, we interleave batches from all six sources during training, inspired by replay strategies from graph continual learning [30]. Table 2 Effect of masking probability on reconstruction MSE loss (test set). The best result is shown in bold. Masking Prob. 0.15 0.25 0.50 1.00
MSE Loss 8.90 × 10−5 9.80 × 10−5 1.19 × 10−4 1.73 × 10−4
We select a masking probability of 0.15 based on a hyperparameter search (Table 2). Lower probabilities yield more stable training and consistently better reconstruction, while higher probabilities tend to oversmooth predictions and diminish the model’s ability to recover fine-grained feature information. The final model uses 256 hidden channels, two shared SAGEConv layers, a batch size 16,384, the Adam optimizer (lr = 1.0 × 10−4 ), and 𝛼 = 0.7. We set 𝛼 = 0.7 to prioritize cosine similarity, which captures directional alignment between embeddings, while retaining MSE as a complementary magnitudesensitive term. This weighting was selected based on preliminary experiments on the validation set. Training largely converges by epoch 5 (Fig. 2), and we report a final test MSE of 8.90 × 10−5 after the full 20-epoch schedule.
4.4. Downstream Adaptation For downstream tasks, new node-type-specific linear layers for the target schema and a lightweight prediction head are always trained from scratch. The shared SAGEConv layers transferred from pretraining are treated in one of two ways: (1) Frozen: the shared SAGEConv weights are fixed, enabling rapid adaptation with minimal overhead; (2) Fine-tuned: the shared SAGEConv weights are updated jointly with the prediction head, allowing the model to specialize its relational reasoning to the new database schema. Both approaches rely on the exact same pipeline: constructing the REG, obtaining BART embeddings, and propagating them through the GNN.
GraphSAGE Pre-training Loss with 0.15 Masking Probability
0.55 0.54 0.53 0.52 0.51
(b) MSE Loss
Training Loss Validation Loss
25
MSE Loss (×10 4)
Scaled Cosine + MSE Loss
(a) Combined Loss (Cosine + MSE) 0.56
0 1
3
5
10
Epoch
15
19
Training Loss Validation Loss
20 15 10 5 0
0 1
3
5
10
Epoch
15
19
Figure 2: Loss curves during GNN pre-training with a masking probability of 0.15. (a) Combined scaled cosine and MSE loss for the training and validation sets. (b) Training MSE loss. The model largely converges by approximately epoch 5.
5. Experiments To evaluate whether our hybrid LM-GNN architecture can generalize to unseen relational databases, we conduct experiments on a held-out dataset from RelBench. We first describe the experimental setup, including the dataset, task, baselines, and training procedure (Section 5.1). We then present and analyze the main results in comparison to supervised methods and relational foundation models (Section 5.2). Finally, we perform an ablation study to isolate the contributions of the BART encoder and the GNN component (Section 5.3).
5.1. Experimental Setup Dataset and Task. We evaluate our framework on the held-out rel-f1 database, which comprises 9 relational tables (including drivers, results, and races) with 97,606 rows. Its small size and distinct domain make it suitable for testing generalization. We focus on the driver-dnf node classification task, predicting, for a given driver identifier and timestamp, whether the driver will fail to finish at least one race within the subsequent one-month window. We report ROC-AUC as the primary metric following RelBench conventions, alongside precision, accuracy, and F1-score in our ablation study. Baselines. We benchmark our framework against several supervised methods: LightGBM [31], manual feature engineering by data scientists [29], and RDL [1], which serves as a supervised, perdatabase GNN baseline. Additionally, we evaluate against recent relational foundation models, including the Griffin-pretrained variant from [3](pre-trained exclusively on single-table datasets, evaluated without RDB-specific fine-tuning), and KumoRFM [4], a commercial model evaluated in both in-context and fine-tuned settings. Baseline scores are sourced from their respective original papers, with the exception of the Griffin-pretrained variant, which we evaluated locally using the official open-source implementation. Given that relational foundation models are only beginning to emerge with scarce public implementations, this evaluation provides a representative snapshot of the current landscape rather than a strictly controlled comparison. Training. Following Section 4.4, all rows are encoded via the fine-tuned BART model to construct the initial REG. We transfer the shared SAGEConv layers from the pre-training phase, while freshly initializing new node-type-specific linear layers for the unseen target schema. In the frozen setting, only these new type-specific layers, the date encoder, and the MLP head are updated; in the fine-tuned setting, the transferred SAGEConv layers are additionally updated. As illustrated in Fig. 3, the target driver node
Features from driver-dnf task Driver
Date
Pre-trained BART
Date Encoder
Pre-trained GNN
Driver Embedding
Date Embedding
Concatenated Embeddings
Label from driver-dnf task
MLP Head
Cross Entropy Loss True Class Label
Prediction (2 Classes)
Figure 3: Downstream adaptation pipeline for the driver-dnf classification task. The driver identifier is encoded by the pre-trained BART encoder and enriched via the pre-trained GNN, while the task date is encoded by a separate date encoder. The resulting embeddings are concatenated and passed through an MLP head trained with cross-entropy loss. Color coding: light red denotes task input features, grey represents learnable or pre-trained network components, blue indicates intermediate embeddings, orange highlights model predictions, and green shows ground-truth labels.
embeddings are enriched through the GNN. In parallel, the task date is processed by a 32-dimensional encoder consisting of two linear layers with ReLU activations. These two representations are then concatenated and fed into an MLP classification head, which comprises three linear layers equipped with ReLU and dropout. The entire downstream architecture is optimized using cross-entropy loss. For this predictive task, we directly utilize the 256-dimensional GNN output rather than projecting it back to 1024 dimensions as was required during pre-training.
5.2. Results and Analysis Table 3 reports the test ROC-AUC scores on the driver-dnf task. Table 3 Test ROC-AUC on the driver-dnf task (rel-f1 dataset). Higher is better. Results of our hybrid model are shown in bold. All baseline scores are taken from the respective publications [3, 4], while the score for the Griffin-pretrained variant was obtained by evaluating the official model checkpoint on our task. Category
Model
ROC-AUC
Supervised
LightGBM Data Scientist RDL
68.86 69.80 72.62
Foundational
KumoRFM (in-context) KumoRFM (fine-tuned) Griffin (pre-trained) Griffin (fine-tuned) Hybrid (frozen) Hybrid (fine-tuned)
82.41 82.63 59.64 70.91 61.40 67.40
Our fine-tuned hybrid model achieves a competitive score of 67.40, approaching the performance of established supervised methods like LightGBM (68.86) and manual feature engineering by data scientists (69.80). These supervised approaches have long been regarded as the gold standard for relational prediction tasks. Furthermore, our fine-tuned model reaches within 5.22 points of the fully supervised, per-database RDL (72.62). This indicates that our hybrid architecture, despite being pretrained on a limited RelBench subset, effectively bridges the gap to heavily optimized, task-specific baselines. The comparison with Griffin is particularly revealing regarding the importance of relational context. While our fine-tuned hybrid model (67.40) remains below Griffin’s fine-tuned variant (70.91), the most notable result emerges in the frozen-backbone transfer setting. Our frozen hybrid model (61.40) explicitly outperforms the pre-trained Griffin variant (59.64). The Griffin-pretrained variant, by design pre-trained exclusively on single-table data without exposure to relational structure, struggles to adapt to the interconnected structure of relational databases. By explicitly pre-training on REGs, our model circumvents this limitation, underscoring how critical relational structure awareness is for downstream performance. KumoRFM (82.41 to 82.63) remains significantly ahead of our approach. However, this gap is expected given KumoRFM’s advanced graph transformers, multi-modal encoders, and vastly larger training scale. Moreover, it is a closed-source commercial product, making direct comparison under identical conditions impossible. We therefore interpret KumoRFM primarily as an empirical upper bound that demonstrates what is achievable with sufficient scale and dedicated resources.
5.3. Ablation Study Table 4 isolates the contribution of each component by varying embeddings (BART vs. random) and GNN usage (none, frozen, fine-tuned). The BART + GNN combination consistently outperforms all other configurations. Without the GNN, BART-only performance drops sharply to an ROC-AUC of 43.90, demonstrating that the MLP head alone cannot compensate for absent relational context. Random embeddings with a fine-tuned GNN reach only 58.00, confirming that BART pre-training provides substantial semantic value beyond what the GNN alone can recover. Fine-tuning the GNN further improves the best configuration from 61.40 to 67.40. Table 4 Ablation results on the test set. "BART" denotes fine-tuned BART embeddings; "Random Emb." denotes embeddings sampled from the same value range as the BART embeddings. "No fine-tuning" keeps GNN weights frozen during downstream training; "with fine-tuning" allows them to be updated. Accuracy and F1 are omitted as they collapse to static values (70.50 and 82.70, respectively) across all configurations due to the majority-class prior (≈ 70.5%). ROC-AUC captures the actual differences in ranking quality. Model
Prec.
ROC-AUC
BART + GNN (no fine-tuning) + Head BART + GNN (with fine-tuning) + Head BART + Head Random Emb. + Head Random Emb. + GNN (no fine-tuning) + Head Random Emb. + GNN (with fine-tuning) + Head
79.60 83.20 68.30 74.20 74.90 76.00
61.40 67.40 43.90 55.40 53.50 58.00
BART embeddings without the GNN (ROC-AUC 43.90) perform worse than random embeddings without the GNN (55.40). We attribute this to a domain shift effect: the BART encoder was fine-tuned on six other RelBench databases and never exposed to rel-f1. Its embeddings may therefore occupy a narrow region of the feature space that is poorly aligned with the downstream task, making it difficult for the MLP head to extract a linearly separable signal. Random embeddings, by contrast, are uniformly distributed and provide a more neutral starting point for the linear operations of the classifier. Crucially,
while BART embeddings preserve relative semantic structure (semantically similar rows lie close to one another), the MLP head relies on linearly separable absolute coordinates and struggles to exploit this latent structure. This relative geometry is exactly the signal that message passing amplifies: once the GNN is introduced, the ordering is reversed (61.40 vs. 53.50 for frozen; 67.40 vs. 58.00 for fine-tuned), confirming that the GNN can propagate and amplify the latent semantic structure that the MLP head alone cannot recover. Due to the severe class imbalance (majority-class prior of ≈ 70.5%), accuracy and F1 computed at the default threshold collapse to static values (70.50 and 82.70, respectively) across all configurations. We therefore omit them from our reported results. While precision reflects minor differences in the sparse positive-class predictions, ROC-AUC, being threshold-independent, most reliably captures the actual differences in ranking quality and serves as our primary metric.
Figure 4: Training and validation metrics (loss, accuracy, and ROC-AUC) on the rel-f1 downstream task with frozen GNN parameters. The validation ROC-AUC exhibits considerable instability and does not show a clear upward trend.
The training dynamics (Figs. 4 and 5) corroborate these findings. With GNN fine-tuning, ROCAUC improves steadily across epochs, indicating that the model progressively adapts its relational representations to the new database. In contrast, the frozen setup produces unstable validation curves without a clear upward trend, suggesting that the pre-trained GNN representations require further adaptation to fully benefit the downstream task. In summary, these results demonstrate that (i) BART pre-training provides strong row-level semantic features, (ii) the GNN effectively enriches them with relational context through message passing, and (iii) fine-tuning the pre-trained GNN is particularly beneficial when adapting to unseen relational databases.
Figure 5: Training and validation metrics (loss, accuracy, and ROC-AUC) on the rel-f1 downstream task with learnable GNN parameters. In contrast to the frozen setting (Fig. 4), validation ROC-AUC improves steadily across epochs.
6. Discussion Building upon our experimental findings, we first analyze the roles of BART as a row encoder and the GNN as a relational encoder, examining how they complement each other. We then address the current limitations of our framework and identify the gaps that must be bridged before it can serve as a true relational foundation model. BART as Row Encoder. Using BART to encode relational rows leverages linguistic pre-training to capture attribute-value dependencies and offers a streamlined and unified alternative to type-specific multi-modal encoders. Fig. 6, produced with t-SNE [32], confirms that the fine-tuned encoder groups same-table rows into coherent clusters, though with some overlap and outliers. However, BART alone is limited to intra-row semantics (ROC-AUC 43.90, Table 4). It inherently lacks the mechanism to explicitly model cross-table dependencies and faces strict sequence length constraints (e.g., a maximum of 1024 tokens), which restricts its applicability to extremely wide tables. These limitations strongly motivate the subsequent GNN stage.
Figure 6: A t-SNE visualization of row embeddings after BART encoding (before GNN processing) for the rel-f1 database. Each color corresponds to a distinct node type within the generated heterogeneous graph (e.g., drivers, races). The encoder produces coherent type-specific clusters, although some expected overlap is visible.
GNN as Relational Encoder. The GNN compensates by injecting relational context through message passing over REGs. As shown in Fig. 7, after GNN propagation the embeddings form tighter, wellseparated per-table clusters with minimal noise, demonstrating that relational context produces more structured representations. The ablation confirms this contribution. Adding the GNN raises ROC-AUC from 43.90 to 61.40 (frozen) or 67.40 (fine-tuned). Trade-offs include computational overhead from message passing over million-node graphs, the risk of oversmoothing with mean aggregation, and the lack of joint LM-GNN training leaving potential cross-modal synergies under-explored. Limitations. Several factors currently prevent this framework from acting as a true relational foundation model. First, the pre-training scale is limited. We used only 100,000 rows for BART and 6 RelBench databases for the GNN, which is substantially smaller than the corpora typically used to train modern foundation models. Second, evaluation is restricted to a single downstream task. Broader
Figure 7: A t-SNE visualization of row embeddings after GNN message passing for the rel-f1 database. The color coding matches that of Fig. 6. Compared to the initial BART embeddings, the GNN propagation yields tighter and more distinctly separated clusters for each node type, visually demonstrating that the injection of relational context produces highly structured representations.
assessment across regression, link prediction, and multiple held-out databases is needed to substantiate generalization claims. Third, our deliberately simple architecture (single LM encoder, GraphSAGE with mean aggregation) trades expressiveness for architectural parsimony compared to Griffin’s multi-modal encoders [3] or KumoRFM’s graph transformers [4]. Fourth, the sequential two-stage training disrupts the gradient flow between the BART encoder and the GNN, preventing the framework from learning a joint distribution across semantic and relational modalities. Fifth, all reported results are based on single experimental runs. While our findings are competitive, establishing statistical significance through multiple trials is paramount [33, 34] within such densely populated and competitive benchmarks.
7. Conclusion and Open Questions We proposed a lightweight hybrid LM-GNN framework combining a fine-tuned BART encoder for intrarow semantics with a GraphSAGE-based GNN for relational context enrichment. On the driver-dnf task from RelBench’s rel-f1 dataset, the model achieved an ROC-AUC of 67.40. This performance comes within 1.46 to 2.40 points of supervised baselines (LightGBM, Data Scientist) and is 5.22 points below RDL. Notably, we achieved these competitive results using a modest budget of 100,000 rows for BART fine-tuning and 6 REGs totaling 28.7 million nodes and 40.1 million edges for GNN pre-training. These results extend the vision of Vogel et al. [5] beyond data engineering tasks, demonstrating that hybrid architectures can serve as a blueprint for general-purpose relational models. Several open questions remain for advancing this framework: (1) Can joint LM-GNN training, for example through end-to-end backpropagation, align semantic and relational representations more effectively than the current sequential pipeline? (2) How does scaling to substantially larger and more diverse pre-training corpora affect downstream performance? (3) Does the architecture generalize to broader task types such as regression and link prediction, and to databases with more complex schemas? While our results already demonstrate the viability of the hybrid LM-GNN paradigm, addressing these questions is critical to transform the current framework from a proof-of-concept into a robust relational foundation model.
Acknowledgments The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Jingcheng Wu. Jingcheng Wu and Ratan Bahadur Thapa have been funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - SFB 1574 - Project number 471687386.
Declaration on Generative AI During the preparation of this work, the authors utilized Gemini for the purpose of a grammar and spelling check. The use of Generative AI was strictly confined to minor linguistic refinements to improve the clarity of the original text.
References [1] M. Fey, W. Hu, K. Huang, J. E. Lenssen, R. Ranjan, J. Robinson, R. Ying, J. You, J. Leskovec, Position: Relational deep learning-graph representation learning on relational databases, in: Forty-first International Conference on Machine Learning, 2024. URL: https://proceedings.mlr.press/v235/fey 24a.html. [2] V. P. Dwivedi, C. Kanatsoulis, S. Huang, J. Leskovec, Relational deep learning: Challenges, foundations and next-generation architectures, in: Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, 2025, pp. 5999–6009. doi:10.1145/3711896.3736 558. [3] Y. Wang, X. Wang, Q. Gan, M. Wang, Q. Yang, D. Wipf, M. Zhang, Griffin: Towards a graph-centric relational database foundation model, in: ICML, volume 267 of Proceedings of Machine Learning Research, PMLR, 2025, pp. 64604–64627. URL: https://proceedings.mlr.press/v267/wang25da.html. [4] M. Fey, V. Kocijan, F. Lopez, J. E. Lenssen, J. Leskovec, KumoRFM: A Foundation Model for InContext Learning on Relational Data, White Paper, Kumo AI, 2025. URL: https://kumo.ai/research /kumo_relational_foundation_model.pdf. [5] L. Vogel, B. Hilprecht, C. Binnig, Towards foundation models for relational databases [vision paper], arXiv preprint arXiv:2305.15321 (2023). doi:10.48550/ARXIV.2305.15321. [6] R. Bommasani, D. A. Hudson, et al., On the opportunities and risks of foundation models, CoRR abs/2108.07258 (2021). doi:10.48550/ARXIV.2108.07258. [7] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Language models are few-shot learners, Advances in Neural Information Processing Systems 33 (2020) 1877–1901. URL: https://proceedings.neurips.cc/paper/2020/hash/1 457c0d6bfcb4967418bfb8ac142f64a-Abstract.html. [8] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, I. Sutskever, Learning transferable visual models from natural language supervision, in: Proceedings of the 38th International Conference on Machine Learning (ICML), volume 139 of Proceedings of Machine Learning Research, 2021, pp. 8748–8763. URL: https://procee dings.mlr.press/v139/radford21a.html. [9] H. Zhou, L. Halilaj, S. Monka, S. Schmid, Y. Zhu, J. Wu, N. Nazer, S. Staab, Seeing and knowing in the wild: Open-domain visual entity recognition with large-scale knowledge graphs via contrastive learning, in: AAAI, AAAI Press, 2026, pp. 13638–13646. doi:10.1609/AAAI.V40I16.38370. [10] L. Ericsson, H. Gouk, C. C. Loy, T. M. Hospedales, Self-supervised representation learning: Introduction, advances, and challenges, IEEE Signal Process. Mag. 39 (2022) 42–62. doi:10.1109/ MSP.2021.3134634. [11] F. Zhuang, Z. Qi, K. Duan, D. Xi, Y. Zhu, H. Zhu, H. Xiong, Q. He, A comprehensive survey on transfer learning, Proc. IEEE 109 (2021) 43–76. doi:10.1109/JPROC.2020.3004555. [12] T. Chen, C. Guestrin, XGBoost: A scalable tree boosting system, in: Proceedings of the 22nd ACM
SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), 2016, pp. 785–794. doi:10.1145/2939672.2939785. [13] V. Borisov, T. Leemann, K. Seßler, J. Haug, M. Pawelczyk, G. Kasneci, Deep neural networks and tabular data: A survey, IEEE Transactions on Neural Networks and Learning Systems 35 (2022) 7499–7519. doi:10.1109/TNNLS.2022.3229161. [14] L. Grinsztajn, E. Oyallon, G. Varoquaux, Why do tree-based models still outperform deep learning on typical tabular data?, Advances in Neural Information Processing Systems 35 (2022) 507–520. URL: http://papers.nips.cc/paper_files/paper/2022/hash/0378c7692da36807bdec87ab043cdadc-A bstract-Datasets_and_Benchmarks.html. [15] X. Huang, A. Khetan, M. Cvitkovic, Z. Karnin, Tabtransformer: Tabular data modeling using contextual embeddings, arXiv preprint arXiv:2012.06678 (2020). doi:10.48550/ARXIV.2012.06 678. [16] Y. Gorishniy, I. Rubachev, V. Khrulkov, A. Babenko, Revisiting deep learning models for tabular data, Advances in Neural Information Processing Systems 34 (2021) 18932–18943. URL: https: //proceedings.neurips.cc/paper/2021/hash/9d86d83f925f2149e9edb0ac3b49229c-Abstract.html. [17] G. Somepalli, M. Goldblum, A. Schwarzschild, C. B. Bruss, T. Goldstein, Saint: Improved neural networks for tabular data via row attention and contrastive pre-training, arXiv preprint arXiv:2106.01342 (2021). doi:10.48550/ARXIV.2106.01342. [18] N. Hollmann, S. Müller, L. Purucker, A. Krishnakumar, M. Körfer, S. B. Hoo, R. T. Schirrmeister, F. Hutter, Accurate predictions on small data with a tabular foundation model, Nature 637 (2025) 319–326. doi:10.1038/s41586-024-08328-6. [19] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, L. Zettlemoyer, BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension, in: ACL, Association for Computational Linguistics, 2020, pp. 7871–7880. doi:10.18653/V1/2020.ACL-MAIN.703. [20] P. Yin, G. Neubig, W. Yih, S. Riedel, Tabert: Pretraining for joint understanding of textual and tabular data, in: ACL, Association for Computational Linguistics, 2020, pp. 8413–8426. doi:10.18653/V1/2020.ACL-MAIN.745. [21] X. Deng, H. Sun, A. Lees, Y. Wu, C. Yu, TURL: Table understanding through representation learning, Proceedings of the VLDB Endowment 14 (2020) 307–319. doi:10.14778/3430915.3430921. [22] N. Tang, J. Fan, F. Li, J. Tu, X. Du, G. Li, S. Madden, M. Ouzzani, RPT: relational pre-trained transformer is almost all you need towards democratizing data preparation, Proc. VLDB Endow. 14 (2021) 1254–1261. doi:10.14778/3457390.3457391. [23] T. N. Kipf, M. Welling, Semi-supervised classification with graph convolutional networks, in: 5th International Conference on Learning Representations (ICLR), 2017. URL: https://openreview.net /forum?id=SJU4ayYgl. [24] Z. Ding, J. Wu, J. Wu, Y. Xia, B. Xiong, V. Tresp, Temporal fact reasoning over hyper-relational knowledge graphs, in: EMNLP (Findings), Findings of ACL, Association for Computational Linguistics, 2024, pp. 355–373. doi:10.18653/V1/2024.FINDINGS-EMNLP.20. [25] W. Hamilton, Z. Ying, J. Leskovec, Inductive representation learning on large graphs, Advances in Neural Information Processing Systems 30 (2017). URL: https://proceedings.neurips.cc/paper/201 7/hash/5dd9db5e033da9c6fb5ba83c7a7ebea9-Abstract.html. [26] Z. Hou, X. Liu, Y. Cen, Y. Dong, H. Yang, C. Wang, J. Tang, Graphmae: Self-supervised masked graph autoencoders, in: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022, pp. 594–604. doi:10.1145/3534678.3539321. [27] V. P. Dwivedi, C. K. Joshi, A. T. Luu, T. Laurent, Y. Bengio, X. Bresson, Benchmarking graph neural networks, Journal of Machine Learning Research 24 (2023) 1–48. URL: https://jmlr.org/papers/v2 4/22-0567.html. [28] M. Fey, J. E. Lenssen, Fast graph representation learning with pytorch geometric, CoRR abs/1903.02428 (2019). doi:10.48550/ARXIV.1903.02428. [29] J. Robinson, R. Ranjan, W. Hu, K. Huang, J. Han, A. Dobles, M. Fey, J. E. Lenssen, Y. Yuan, Z. Zhang, et al., Relbench: A benchmark for deep learning on relational databases, Advances in Neural
Information Processing Systems 37 (2024) 21330–21341. URL: http://papers.nips.cc/paper_files/p aper/2024/hash/25cd345233c65fac1fec0ce61d0f7836-Abstract-Datasets_and_Benchmarks_Track .html. [30] X. Zhang, D. Song, D. Tao, Continual learning on graphs: Challenges, solutions, and opportunities, arXiv preprint arXiv:2402.11565 (2024). doi:10.48550/ARXIV.2402.11565. [31] G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, T.-Y. Liu, Lightgbm: A highly efficient gradient boosting decision tree, Advances in Neural Information Processing Systems 30 (2017). URL: https://proceedings.neurips.cc/paper/2017/hash/6449f44a102fde848669bdd9eb6b76fa-Abs tract.html. [32] L. van der Maaten, G. Hinton, Visualizing data using t-SNE, Journal of Machine Learning Research 9 (2008) 2579–2605. URL: https://www.jmlr.org/papers/v9/vandermaaten08a.html. [33] Y. Zhu, N. Potyka, M. Nayyeri, B. Xiong, Y. He, E. Kharlamov, S. Staab, Predictive multiplicity of knowledge graph embeddings in link prediction, in: EMNLP (Findings), Findings of ACL, Association for Computational Linguistics, 2024, pp. 334–354. doi:10.18653/V1/2024.FINDI NGS-EMNLP.19. [34] Y. Zhu, J. Wu, Y. Wang, H. Zhou, J. Chen, E. Kharlamov, S. Staab, Certainty in uncertainty: Reasoning over uncertain knowledge graphs with statistical guarantees, in: EMNLP, Association for Computational Linguistics, 2025, pp. 8730–8752. doi:10.18653/V1/2025.EMNLP-MAIN.441.