Incremental Evaluation and Training in Relational Deep Learning

Incremental Evaluation and Training in Relational Deep Learning

Jakub Peleška Czech Technical University in Prague jakub.peleska@cvut.cz

Gustav Šír Czech Technical University in Prague gustav.sir@cvut.cz

arXiv:2608.13023v1 [cs.LG] 13 Aug 2026

Abstract Relational Deep Learning (RDL) models multi-tabular databases as temporal heterogeneous graphs to enable end-to-end representation learning. However, prevailing RDL evaluation practices rely on static, single-episode dataset snapshots, overlooking the continuous, time-evolving nature of real-world databases. Consequently, current RDL benchmarks fail to capture how model performance changes as new data accumulates over time. To address this limitation, we introduce an incremental, multi-episode evaluation and training paradigm to assess and improve the temporal robustness and adaptability of state-of-the-art RDL models. Using established large-scale datasets, we examine data evolution and model training dynamics, demonstrating that temporal concept drifts occur in the majority of predictive tasks. We present multiple incremental training regimes for fine-tuning the models and demonstrate that transfer learning is both feasible and highly effective in the RDL setting. Alongside a new temporal evaluation metric that prioritizes near-future accuracy, we show that our incrementally fine-tuned models consistently outperform the standard, expensive, from-scratch trained baselines.

1

Introduction

Relational databases (RDBs) are the foundational infrastructure for data storage across a vast majority of modern industries [6, 1, 38]. Despite their ubiquity and immense predictive value, applying traditional machine learning (ML) to RDBs presents significant challenges. The inherently heterogeneous, dynamically structured nature of relational data—spanning multiple interconnected tables with varying schemas—resists the fixed-size feature vector representations required by classical ML algorithms. Consequently, practitioners are often forced into laborious manual feature engineering or lossy table joins that discard important structural semantics [11]. To address these bottlenecks, Relational Deep Learning (RDL) [41, 11] has recently emerged as a transformative ML paradigm. By interpreting an entire relational database as a heterogeneous graph, RDL leverages Graph Neural Networks (GNNs) [35, 2] alongside deep tabular representation learning [13, 30] to enable end-to-end predictive modeling directly on the native structures of the RDBs (Figure 1). While RDL successfully eliminates the need for manual feature flattening, bringing ML significantly closer to the raw, heterogeneous reality of relational data, it currently neglects a foundational dimension of real-world RDBs: their continuous temporal evolution. Indeed, databases are not static repositories; they dynamically expand and shift over time, causing models trained on historical snapshots to become incrementally obsolete. Despite this dynamic reality, current RDL evaluation practices remain focused on static models. Given the structural complexity of RDBs, which prevents classic i.i.d. sample-based predictions, predictive tasks in RDL are defined by augmenting the schema with a dedicated “task table” that specifies target entities and labels. To avoid temporal leakage, existing frameworks [33, 29] correctly use timestamps to perform chronological train–test splits [11, 32]. However, this snapshot-based evaluation assesses models in a single, isolated episode, fundamentally failing to capture how model performance degrades over time, e.g., due to Preprint.

Heterogeneous Relational Entity Graph

Relational Database

Update on ∆Gi

Model i (θi )

Updated i2 Items

I_0 | U_2 | T0

u1

Weights

i2

Users

Msg Pass

Task Head

FFN

U_1 | 25 U_2 | 31

Σ MHA

i0

u2

i3

Transformer

New ∆Gi

GNN

Past Rows

I_2 | U_1 | T2 I_3 | U_2 | T3

New Links

Table Encoder

Logs

Model i−1 (θi−1 )

Append

L_1 | ...

MLP

Leval (θi )

I_1 | U_1 | T1

i1

New Rows

New ∆Gi

Old Gt−1

P_1 | ...

Interactions

i3

Past Links

Past Gt−1 u1 u2 i1

i0

Ltrain (θi−1 ) Loss

(a) Relational DB Increment

(b) Temporal Graph Evolution

(c) Incremental RDL Fine-Tuning

Figure 1: Incremental RDL workflow with (a) append-only data updates, (b) induced relational graph increments ∆Gi on top of historical context Gt−1 , and (c) episode-wise fine-tuning from θi−1 to θi . temporal concept drift. Characterizing this temporal stability and evaluating how models adapt to new increments of relational data then remains an unaddressed prerequisite for advancing toward robust, real-world RDL foundation models [32, 36]. Contributions. To address the gap between static benchmarking and the dynamic reality of relational data, we introduce an incremental, multi-episodic evaluation and training paradigm, visualized in Figure 1. Tailored for robust assessment of RDL models, our approach simulates the real-world life-cycle of database growth by transitioning RDL benchmarking from static snapshots to continuous, horizon-shifting evaluation. Leveraging state-of-the-art models and large-scale datasets [33, 29, 15], we empirically demonstrate that temporal concept drift is a prevalent factor in standard RDL tasks. To combat model degradation, we define and benchmark multiple incremental fine-tuning strategies, showing that knowledge transfer is highly effective and computationally superior to from-scratch retraining in the continuous RDL setting. Finally, we propose a suitable exponential decay-based evaluation that prioritizes near-future predictive accuracy, closely aligning benchmark evaluation with practical operational priorities.

2

Related Work

RDL Benchmarks and Frameworks Standardized ecosystems have accelerated RDL research. RelBench [33, 15] formalized predictive tasks with chronological splits to prevent temporal leakage [22]. Concurrently, ReDeLEx [29] expanded the evaluation landscape across a large set of diverse databases to analyze structural impacts on model performance. While these frameworks provide essential building blocks for static testing, they lack the multi-episodic evaluation necessary to measure continuous model degradation and real-world lifecycle dynamics. Architectural Advances and Foundation Models To capture the extreme heterogeneity of RDBs, many recent methods integrate tabular representation learning [25, 30], LLMs [39], and graph transformers [9, 24]. This progress naturally steers the field toward RDL foundation models [37, 10, 12, 19, 36, 32], supported by task-agnostic pretraining [31] and synthetic data generation [20, 23]. While these foundational architectures are largely designed for static deployment, the overarching shift towards foundation models and transfer learning strongly overlaps with our proposed temporal fine-tuning, forming a prerequisite for maintaining their robustness over time. Temporal Graph Representation Learning Relational data evolution conceptually intersects with Temporal Graph Representation Learning [21]; nevertheless, traditional temporal GNNs (e.g., TGN [34], TGAT [8]) model high-frequency dynamic systems to predict immediate interactions. In contrast, our incremental RDL paradigm does not treat databases as stateful time-series systems. Instead, we focus on macro-level volumetric growth and long-term temporal concept drift to assess model obsolescence over extended horizons. Continual Learning and Concept Drift By addressing model obsolescence, our approach conceptually bridges RDL with Continual Learning (CL) [28] and concept drift adaptation [26]. Standard CL and Continual Graph Learning [42] mitigate catastrophic forgetting primarily in homogeneous 2

structures. RDL, however, involves highly heterogeneous, multi-tabular data increments. Our work provides the first formal benchmark to evaluate CL strategies directly within the dynamically evolving environment of real-world databases.

3

Relational Deep Learning

RDL is a machine learning paradigm built on interpreting relational databases as heterogeneous graphs. This perspective enables the direct application of deep learning techniques—specifically GNNs alongside tabular representation learning—on the native structures of RDBs. While the underlying idea was explored in earlier work [7, 41], it was recently formalized under the RDL umbrella [11] to consolidate these efforts into a cohesive research field, detailed below. 3.1

Graph Representation

Formally, a relational database R is defined as a finite set of relations R1 , R2 , . . . , Rn . Each relation R consists of a heading (signature) R/n , formed by the set of its attributes, and a body, formed by the tuples of attribute values ti = (a1 , a2 , . . . , an ), commonly represented as a table TR . Additionally, the relation R can be defined extensionally by the unordered set of its tuples: R = {t1 , t2 , . . . , tm }, corresponding to the rows (or entities) of the table TR . Practically, the attributes forming the rows of each table TR are categorized into four functional groups: (i) primary key (PK), a minimal set of attributes P KR that uniquely identifies each entity, such that for any two distinct tuples ta , tb ∈ R, ta [P KR ] ̸= tb [P KR ]; (ii) foreign key (FK), a set of attributes in a relation Ri that references the primary key of another relation Rj , establishing a structural dependency ensuring referential integrity ∀t ∈ Ri , t[F KRi →Rj ] ∈ {t′ [P KRj ] | t′ ∈ Rj }; (iii) feature attributes, containing characteristic information about the entity; and, optionally, (iv) a timestamp, indicating when the row was created or last modified. The relational entity graph of a database R is a temporal heterogeneous graph defined as G = (V, E, ϕ, ψ, τ ). The set of nodes V is defined as the union of all tuples from each relation: V = {vi,j | Ri ∈ R, tj ∈ Ri }, inferring vi,j ≃ tj ∈ Ri . Consequently, the set of edges E is defined by the PK-FK relationships: E = {(vi,k , vj,l ) | tk ∈ Ri , tl ∈ Rj , tk [F KRj ] = tl [P KRj ] ∨ tl [F KRi ] = tk [P KRi ]} (Figure 1b). To capture the heterogeneity of the database, the function ϕ : V → T v serves as a node mapping. The set of node types T v corresponds directly to the set of relations within R, meaning ϕ maps a node vi,j strictly to its original source relation Ri . Similarly, ψ : E → T e is an edge mapping function, where the set of edge types T e corresponds to the specific pairs of PK and FK attributes (or their reversed counterparts) that generated the edge. Finally, the time mapping function τ : V → T assigns a timestamp to each node such that τ (v) = Tv . If a temporal attribute is not available for a given row, a zero timestamp is assigned by default. 3.2

Neural Methods

Building upon the graph representation G, RDL models typically follow a four-stage pipeline: (i) (0) (0) a table-level attribute encoder initializes node embedding matrices hv ∈ Rd ×n by encoding each feature attribute A1 , . . . , An of the node’s source relation ϕ(v) based on its semantic data type; (ii) a table-level tabular model employs tabular architectures [4, 18] to refine these embeddings (l)

(l)

(l)

(l)

d

into hv ∈ Rd ×n , optionally compressing the matrix into a single vector hv ∈ R ϕ(v) ; (iii) a graph neural model applies message-passing based on the defined edge types T e , using standard GNNs [16, 35, 2] for vectors or custom schemes [30, 5] for attribute-matrices; and (iv) a task-specific model head maps the final node embeddings to predictions, typically via simple MLP layers. This RDL architectural blueprint is illustrated in Figure 1c. 3.3

Predictive Tasks

Predictive tasks in RDL are formalized by constructing a dedicated “task table” Ttask that extends the existing relational database R [11, 22]. A task table Ttask consists of tuples defining the individual prediction instances, abstracted as (v, y, t). Here, v ∈ V is the target entity (node) identified via a 3

(0)

Users U_1 | Age:

25

I_1 | U_2 | P_1 | ti−1

U_2 | Age:

31

I_2 | U_1 | P_2 | ti

U_3 | Age:

42

I_3 | U_3 | P_1 | ti+1

U_4 | Age:

29

I_4 | U_1 | P_1 | ti+2 ⇒ y

U_5 | Age:

35

I_5 | U_2 | P_2 | ti+2 ⇒ y

Ep. 0

t ≤ ti+2

...

A

I_6 | U_3 | P_1 | tval

P_2 | Cat:

B

I_7 | U_5 | P_4 | tval +1

P_3 | Cat:

C

P_4 | Cat:

D

(1)

...

∆w ti

ti−1

t ∈ ∆w

Ep. 1

Co nt ex t

(1)

tval

...

∆w ti+1

(1)

P_1 | Cat:

∆w

VAL

Q(ti−1 ) Q(ti ) TRAIN

I_8 | U_4 | P_3 | ti+2 ⇒ y

Products

tval

TRAIN

Interactions

Ta rg

et

(1)

t > tval

∆w

VAL ...

ti+2

Q(ti+1 ) SQL

(1)

I_9 | U_4 | P_3 | tval +2

Q(ti+2 )

Figure 2: Left: a relational task table is chronologically split by anchor times, with a leakage barrier (red) separating validation from training context, further split into respective target task windows of size ∆w, induced by the predictive queries Q(ti ). Right: evaluation proceeds over increments (i) of length ∆I, shifting the train/validation horizon tval and instantiating the queries at successive anchors, while extracting context G≤t and future targets from the newly revealed task windows. foreign key reference Ttask [F K], y ∈ Y is the ground-truth target label to be predicted (e.g., y ∈ R for regression), and t ∈ T is the anchor time, specifying the moment at which the prediction is made. The objective of an RDL model fθ is to predict the target label ŷ = fθ (Gv,≤t ) ≈ y, where Gv,≤t is the relational subgraph defining the computational context for entity v, strictly filtered to only include information available at or before anchor time t. Depending on the role of t, these tasks are categorized into static and temporal formulations. Static Tasks. Static tasks involve predicting missing values or properties within a database viewed as a single, fixed snapshot in time. The temporal dimension is effectively ignored, reducing instances to pairs (v, y). The objective simplifies to learning a function ŷ = fθ (Gv ), utilizing the entire available graph G without risk of temporal information leakage. In these cases, the target label y belongs to a feature column that already structurally exists within the database but is masked out. Temporal Tasks. Conversely, temporal tasks (or “forecasting tasks”) predict future behaviors or events that have not yet occurred. Here, the anchor time t is crucial, as the label y represents an aggregation or occurrence in a future window (t, t + ∆w] (e.g., total sales over the next 30 days). The set of task labels is formed as Y = {Q∆w (t) | t ∈ Ttask }, where Q is a future-pointing task query 1 with a fixed window size ∆w, and Ttask is a set of task anchor timestamps (see Figure 2). To prevent temporal data leakage, the model must strictly condition its representations on past data, requiring the graph to be temporally masked to yield G≤t = (V≤t , E≤t ), where V≤t = {u ∈ V | τ (u) ≤ t}. Data Splits. The strategy for partitioning data varies fundamentally between the tasks. For static tasks, test instances (v, y) can be split randomly. Temporal tasks, however, necessitate strict chronological evaluation to prevent backward data leakage (Figure 2). These tasks further define preset validation and test timestamps: the training set contains instances where t < tval , the validation set where tval ≤ t < ttest , and the test set where t ≥ ttest . This ensures the model is evaluated exclusively on unseen future time periods.

4

Incremental Evaluation Paradigm

Standard evaluation of models on relational databases relies on static dataset snapshots, failing to capture the dynamic evolution of real-world data where temporal increments dictate both target creation and predictive tasks. To standardize the measurement of model robustness against undetected data shifts and temporal concept drifts, we propose an incremental evaluation paradigm that simulates the real-world deployment lifecycle (Figure 1). This enables researchers to assess how well RDL models adapt to continuously growing databases over time. 1 The queries are implemented through a SQL [3] representation of the task.

4

4.1

Multi-Episodic Relational Tasks

Expanding upon the single-episode temporal tasks defined by future-pointing queries Q∆w (t) (Section 3.3), we introduce a fixed time increment ∆I ≥ ∆w representing the temporal stride between consecutive model updates (Figure 2), where an episode constitutes a batch of new target data arriving over this increment. To transition to multi-episodic setting, we evaluate and update models incrementally over a sequence of monotonically increasing evaluation horizons: T∆I = {ti | ti = ti−1 + ∆I, ti ≤ tval , i ≥ 1}. At each episode i, new training examples generated by Q∆w (t) from newly available anchor times t ∈ (ti−1 , ti ] are revealed up to the anchor time ti . This interval may contain multiple base task windows ∆w. Once evaluated, the true labels become available, and the increment is appended to the historical training data. This strict append-only growth (Figure 1a) prevents temporal leakage, ensuring the model never accesses information beyond its current validation horizon. 4.2

Incremental Training Regimes

As the prediction horizon shifts from episode i − 1 to episode i (Figure 2), a new data increment becomes available. The training set correspondingly expands to include all data up to the new (i) validation timestamp tval . The validation and test datasets shift forward chronologically, maintaining their structural role but evaluating the model on the newly established horizon. Because an increment can span multiple task windows, these shifted splits dynamically adapt to test the model on the exact block of newly revealed target data. To evaluate how effectively models assimilate these increments, we benchmark four training regimes: (i)

1. From Scratch (Baseline): Trained from scratch using all available data up to tval , serving as a (computationally expensive) baseline that follows standard single-episode RDL training. 2. Finetune (Cumulative): Retains weights from episode i − 1 and fine-tunes on the entire (i) historical dataset up to tval . 3. Finetune (Incremental): Updates the episode i − 1 model exclusively on the newest data increment to maximize training speed and immediate responsiveness. 4. Finetune (Upsampled): A hybrid strategy fine-tuning on a distribution that upsamples the most recent increment while retaining historical data to mitigate catastrophic forgetting. This is parameterized by a probability governing how often new increment data is drawn. 4.3

Time-Weighted Evaluation Metrics

In multi-episodic settings, near-future predictions are typically more operationally critical than longterm forecasts. Standard evaluations, however, weight all future samples equally, which can mask a model’s responsiveness to immediate temporal shifts. To prioritize near-future accuracy, we introduce a time-weighted exponential decay applicable to any standard metric (e.g., AUROC, MAE). Given N future evaluation windows ordered by temporal distance from the training horizon (where i = 1 is the immediate next window) and a decay rate α ∈ [0, 1], we assign a normalized weight Ws,i to each sample s in window i: Ws,i = PN

(1 − α)i−1

j=1 |Sj |(1 − α)

j−1

(1)

where |Sj | is the number of samples in window j. This reduces to a uniform mean when α = 0, while α → 1 strictly isolates performance on the immediate next increment.

5

Experiments

We evaluate our multi-episodic paradigm using 12 tasks (both regression and binary classification) defined on 4 different databases from the established RelBench [33] benchmark, which currently serves as the standard for evaluating RDL methods [24, 40]. Through these experiments, we (i) 5

Dataset

Task

Original Data Splits

Total N of ∆

Avg. Samples

Span in days

Train

Val

Test

∆w

∆I

∆w

∆I

∆w

∆I

rel-avito

ad-ctr user-clicks user-visits

5.1k 59.5k 86.6k

1.8k 21.2k 30.0k

1.8k 48.0k 36.1k

5 5 5

4 4 4

1.7k 20.2k 29.2k

2.3k 26.9k 38.9k

4 4 4

6 6 6

rel-f1

driver-position driver-dnf driver-top3

7.5k 11.4k 1.4k

499 566 588

760 702 726

310 475 116

13 13 4

28 27 23

663 998 647

77 48 60

1812 1814 1928

rel-stack

post-votes user-badge user-engagement

2.45M 3.39M 1.36M

156k 247k 85.8k

161k 255k 88.1k

48 42 48

20 19 20

55.5k 88.6k 30.8k

137k 202k 76.1k

91 91 91

225 207 225

rel-trial

site-success study-adverse study-outcome

151k 43.3k 12.0k

19.7k 3.6k 960

22.6k 3.1k 825

21 21 21

9 9 9

8.6k 2.3k 648

21.4k 5.9k 1.6k

365 365 365

912 912 912

Table 1: Temporal and evaluation statistics across datasets and tasks [33] used in the experiments. We report the original split sizes, alongside the total number, average sample count, and temporal span (in days) for both the evaluation windows (∆w) and increments (∆I). Large sample counts are abbreviated (k = thousands, M = millions).

analyze the temporal data distribution, (ii) confirm the presence of temporal concept drift, (iii) evaluate the effectiveness of knowledge transfer via incremental fine-tuning, and (iv) test the time-weighted metrics prioritizing near-future accuracy. 5.1

Incremental Data Analysis

Expanding upon the single-episode temporal tasks defined by future-pointing queries Q∆w (ti ) (Section 3.3 and Figure 2), our multi-episodic paradigm incrementally evaluates models using a fixed time increment ∆I (Section 4.1). Table 1 details the temporal increments (∆w, ∆I) and the original split sizes. Notably, we set the time increment ∆I based on the temporal span of the original validation split, [tval , ttest ), facilitating seamless comparison with the original single-episode setting [33]. The table also indicates the number of available training episodes. We note that, because the final two increments correspond to the original validation and test splits and are excluded from training, tasks on the rel-avito dataset (due to short temporal coverage) and the rel-f1 driver-top3 task (due to the chosen increment span) yield only two training episodes. To detail the temporal data distribution, Figure 3a 2 presents new sample arrivals per time window ∆w alongside cumulative task sizes. From the evolution of task sizes, we identify two principal growth patterns: roughly linear (rel-avito ad-ctr, rel-f1 driver-position) and exponential (rel-stack user-engagement, rel-trial study-adverse). Figure 3b illustrates the evolving cumulative target-value distributions. Notably, with the exception of rel-avito ad-ctr (which spans a brief interval), all remaining tasks exhibit clear evidence of temporal concept drift. 5.2

Experimental Setup

We implement our incremental evaluation and training framework using PyTorch, PyTorch Lightning, and PyTorch Geometric, building upon the RelBench [33] and REDELEX [29] libraries. Model Architecture and Initialization. We employ a standard RDL benchmark architecture [33, 9]: tabular ResNet [14] node encoders (via PyTorch Frame [17]) paired with GraphSAGE [16]. The core GNN uses 2 message-passing layers (hidden dimensionality 128, sum aggregation), followed by task-specific prediction heads with batch normalization. 2We report a subset of tasks in the main-text figures; comprehensive results are in Appendix A.

6

(b) Cumulative value of target y over the full range off the task windows ∆w. Mean value with standard deviation for regression tasks, positive class rate for the binary classification tasks.

(a) Number of new samples arriving in each time window ∆w, together with the cumulative task size, highlighting differences in growth across successive windows in different datasets.

Figure 3: Visual analysis of data distribution over time windows ∆w from anchor timestamps ti .

Graph Construction and Sampling. Following the RDL blueprint, databases are processed into temporal heterogeneous PK-FK graphs (Section 3). To handle large scales, we use mini-batch training with uniform temporal neighbor sampling, restricting context to edges and nodes valid at or prior to the anchor timestamp to strictly prevent temporal leakage (Figure 2). We sample a maximum of 32 neighbors for the first GNN layer and 16 for the second (decaying by a factor of two). For the Finetune (Upsampled) regime, a composed data loader samples mini-batches equally (50% probability) from the newest data increment and the historical data (Figure 1c). Training and Optimization. Models are optimized via Adam (learning rate 0.001) with a batch size of 128. During each episodic increment, models train for a maximum of 2000 steps, artificially limited to 100 batches per epoch to ensure frequent validation checks. All configurations are evaluated across 5 random seeds, reporting the mean and standard deviation. Further compute, runtime, and orchestration details can be found in Appendix A.1.

Figure 4: Performance of the From Scratch models trained in the multi-episodic setting, reported on the original validation and test splits. Metrics are Mean Absolute Error (MAE; lower indicates better performance) for regression tasks and AUROC (higher indicates better performance) for classification.

5.3

Investigating Temporal Concept Drift

To confirm that the observed distribution shifts constitute temporal concept drift, we incrementally train baseline From Scratch models and assess them on the original validation and test splits. Con7

(b) Model metrics after the first 100 training steps.

(a) Top model metrics after complete training.

Figure 5: Performance of different model training regimes in the multi-episodic setting, reporting the validation score for the subsequent increment. Metrics are, again, MAE (lower indicates better performance) and AUROC (higher indicates better performance).

cretely, given anchor times T∆I = {ti | ti = ti−1 + ∆I, ti ≤ tval },3 each model Mi is trained up to ti , with the best checkpoint selected via validation on the subsequent increment (up to ti+1 ). Figure 4 then substantiates the presence of concept drift through a general trend of performance degradation over time across tasks. A minor exception to that trend is rel-f1 driver-position, which exhibits a substantial drop in performance, likely attributable to its relatively small dataset size. Nevertheless, the overall empirical validation of the drift confirms the original motivation for the RDL focus on temporal dynamics. 5.4

Evaluating Fine-tuning Regimes

Our multi-episodic setting enables RDL methods that exploit incrementally expanding data. Figure 5a compares the top performances of our three fine-tuning regimes (Section 4.2) against a From Scratch baseline. The fine-tuning regimes consistently match or outperform the baseline, an effect most pronounced in the rel-stack user-engagement and rel-trial study-adverse tasks. To further illustrate knowledge transfer capabilities, Figure 5b shows model performance after just 100 training steps, resembling a Few-Shot Learning scenario. Notably, pretrained models quickly achieve performance on par with or superior to a fully trained from scratch baseline, underscoring the computational efficiency and predictive potential of continuously fine-tuned models. 5.5

Decayed Evaluation Metrics

To account for the operational prioritization of near-future predictions, Table 2 applies our temporally decayed metrics (Section 4.3) to the regression tasks. A decay of 0.0 yields a standard uniform average, while 0.9 places almost all weight on the immediate next window. Notably, except for driver-position, varying the decay factor preserves the optimal training regime rankings. Nevertheless, absolute metric values shift in distinct patterns: post-votes and ad-ctr exhibit minimal variation, study-adverse shows a consistent performance drop, and driver-position improves at α = 0.3—a variation we suspect reflects differing rates of localized temporal volatility across datasets, with a detailed analysis left for future work. 3t val without a superscript refers to the original (non-incremental) dataset split boundary

8

Dataset

Task

Training Regime - MAE ↓

Decay

FT Cuml.

FT Incr.

FT Upspl.

Scratch

rel-avito

ad-ctr

0.0 0.3 0.9

0.037 0.037 0.035

0.037 0.037 0.036

0.039 0.039 0.037

0.039 0.038 0.037

rel-f1

driver-position

0.0 0.3 0.9

3.256 2.858 3.466

3.408 2.922 3.293

3.278 3.058 3.569

3.106 2.827 3.358

rel-stack

post-votes

0.0 0.3 0.9

0.061 0.061 0.059

0.062 0.062 0.060

0.062 0.062 0.060

0.066 0.065 0.063

0.0 0.3 0.9 0.0 0.3 0.9

0.428 0.431 0.440 46.369 46.565 47.239

0.395 0.396 0.399 45.996 46.163 46.735

0.419 0.420 0.421 46.545 46.819 47.760

0.393 0.392 0.389 48.143 48.281 48.755

site-success rel-trial study-adverse

Table 2: Performance of individual training regimes, trained on the complete training split, assessed using the temporally decayed MAE over the combined original validation and test splits.

6

Conclusion

In this work, we addressed a major shortcoming in standard Relational Deep Learning (RDL) evaluation: the reliance on static dataset snapshots, which fundamentally misaligns with the continuous, dynamic evolution of real-world databases. By formalizing an incremental, multi-episodic evaluation and training paradigm, we provide a framework that accurately reflects the real-world lifecycle of database growth and model deployment. Our empirical analysis across large-scale relational datasets establishes several core findings. First, we validated the widespread prevalence of temporal concept drift in standard RDL tasks, highlighting why snapshot-based testing is insufficient. Furthermore, we showed that incremental fine-tuning strategies successfully mitigate this degradation. Transferring knowledge from previous training episodes consistently matched or exceeded the performance of expensive from-scratch retraining, often adapting within just a few optimization steps. Finally, introducing a time-weighted exponential decay metric allowed for a more realistic assessment of a model’s utility by properly prioritizing near-future predictive accuracy. Altogether, the proposed paradigm and associated training regimes lay the groundwork for transitioning RDL research from isolated benchmarks toward practical, continually learning systems.

Limitations. While our experimental setup currently focuses on append-only database growth, real-world systems often involve data updates and deletions (CRUD operations) [6]. Although our multi-episodic paradigm can in principle accommodate these mutations—such as by dynamically updating historical node states or utilizing temporal edges [21]—doing so introduces complex attribute dependencies that require further formal investigation. Furthermore, while we establish foundational fine-tuning regimes, mitigating catastrophic forgetting over extended horizons remains a challenge [42]. Although the incremental fine-tuning offers a highly scalable alternative as cumulative data grows, optimally balancing this computational efficiency with long-term knowledge retention may require integrating more advanced continual learning techniques, such as dynamic replay buffers or weight regularization [28]. Finally, extending this continuous evaluation to emerging monolithic RDL foundation models [37, 19, 32, 36] remains an open challenge to fully understand their specific resilience to temporal concept drift. 9

Acknowledgments and Disclosure of Funding References [1] R. Agrawal, A. Somani, and Y. Xu. Storage and querying of e-commerce data. In VLDB, volume 1, pages 149–158, 2001. [2] S. Brody, U. Alon, and E. Yahav. How attentive are graph attention networks? In International Conference on Learning Representations, 2022. [3] D. D. Chamberlin and R. F. Boyce. SEQUEL: A structured English query language. In Proceedings of the 1974 ACM SIGFIDET (now SIGMOD) workshop on Data description, access and control, SIGFIDET ’74, pages 249–264, New York, NY, USA, 1974. Association for Computing Machinery. [4] K.-Y. Chen, P.-H. Chiang, H.-R. Chou, T.-W. Chen, and D. T.-H. Chang. Trompt: towards a better deep neural network for tabular data. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org, 2023. [5] T. Chen, C. Kanatsoulis, and J. Leskovec. RelGNN: Composite message passing for relational deep learning. In Forty-second International Conference on Machine Learning, 2025. [6] E. F. Codd. A relational model of data for large shared data banks. 13(6):377–387, June 1970.

Commun. ACM,

[7] M. Cvitkovic. Supervised learning on relational databases with graph neural networks. arXiv preprint arXiv:2002.02046, 2020. [8] C. R. Da Xu, E. Korpeoglu, S. Kumar, and P. Awasthi. Inductive representation learning on temporal graphs. In International Conference on Learning Representations, 2020. [9] V. P. Dwivedi, S. Jaladi, Y. Shen, F. Lopez, C. I. Kanatsoulis, R. Puri, M. Fey, and J. Leskovec. Relational graph transformer. In Temporal Graph Learning Workshop @ KDD 2025, 2025. [10] V. P. Dwivedi, C. Kanatsoulis, S. Huang, and J. Leskovec. Relational deep learning: Challenges, foundations and next-generation architectures. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2, KDD ’25, page 5999–6009, New York, NY, USA, 2025. Association for Computing Machinery. [11] M. Fey, W. Hu, K. Huang, J. E. Lenssen, R. Ranjan, J. Robinson, R. Ying, J. You, and J. Leskovec. Position: Relational deep learning - graph representation learning on relational databases. In Forty-first International Conference on Machine Learning, 2024. [12] M. Fey, V. Kocijan, F. Lopez, J. E. Lenssen, and J. Leskovec. KumoRFM: A Foundation Model for In-Context Learning on Relational Data. May 2025. [13] Y. Gorishniy, I. Rubachev, V. Khrulkov, and A. Babenko. Revisiting deep learning models for tabular data. In Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS ’21, pages 18932–18943, Red Hook, NY, USA, 2021. Curran Associates Inc. [14] Y. Gorishniy, I. Rubachev, V. Khrulkov, and A. Babenko. Revisiting deep learning models for tabular data. In Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS ’21, pages 18932–18943, Red Hook, NY, USA, 2021. Curran Associates Inc. [15] J. Gu, R. Ranjan, C. Kanatsoulis, H. Tang, M. Jurkovic, V. Hudovernik, M. Znidar, P. Chaturvedi, P. Shroff, F. Li, and J. Leskovec. RelBench v2: A Large-Scale Benchmark and Repository for Relational Data, Feb. 2026. arXiv:2602.12606 [cs]. [16] W. Hamilton, Z. Ying, and J. Leskovec. Inductive Representation Learning on Large Graphs. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. 10

[17] W. Hu, Y. Yuan, Z. Zhang, A. Nitta, K. Cao, V. Kocijan, J. Leskovec, and M. Fey. Pytorch frame: A modular framework for multi-modal tabular learning. arXiv preprint arXiv:2404.00776, 2024. [18] X. Hu, W. Tang, C.-K. Hsieh, and S. Shi. Tabtransformer: Tabular data modeling using contextual embeddings. arXiv preprint arXiv:2012.06678, 2020. [19] V. Hudovernik, F. López, V. Kocijan, A. Nitta, J. E. Lenssen, J. Leskovec, and M. Fey. KumoRFM-2: Scaling Foundation Models for Relational Learning, Apr. 2026. arXiv:2604.12596 [cs]. [20] M. Jurkovic, V. Hudovernik, and E. Štrumbelj. Syntherela: A benchmark for synthetic relational database generation. In Will Synthetic Data Finally Solve the Data Access Problem?, 2025. [21] S. M. Kazemi, R. Goel, K. Jain, I. Robertson, H. Schmid, S. Srinivasan, S. Kieffer, Y. Chen, and P. Poupart. Representation learning for dynamic graphs: A survey. The Journal of Machine Learning Research, 21(1):2648–2718, 2020. [22] V. Kocijan, J. Sunil, J. E. Lenssen, V. Deb, X. Xe, F. R. Gomez, M. Fey, and J. Leskovec. Predictive Query Language: A Domain-Specific Language for Predictive Modeling on Relational Databases, Feb. 2026. arXiv:2602.09572 [cs]. [23] V. Kothapalli, R. Ranjan, V. Hudovernik, V. P. Dwivedi, J. Hoffart, C. Guestrin, and J. Leskovec. PluRel: Synthetic Data unlocks Scaling Laws for Relational Foundation Models, Feb. 2026. arXiv:2602.04029 [cs]. [24] D. Lachi, M. Mohammadi, J. Meyer, V. Arora, T. Palczewski, and E. L. Dyer. RGP: A Cross-Attention based Graph Transformer for Relational Deep Learning. Oct. 2025. [25] V. Lachi, A. Longa, B. Bevilacqua, B. Lepri, A. Passerini, and B. Ribeiro. Boosting Relational Deep Learning with Pretrained Tabular Models, Apr. 2025. arXiv:2504.04934 [cs]. [26] J. Lu, A. Liu, F. Dong, F. Gu, J. Gama, and G. Zhang. Learning under concept drift: A review. IEEE transactions on knowledge and data engineering, 31(12):2346–2363, 2018. [27] P. Moritz, R. Nishihara, S. Wang, A. Tumanov, R. Liaw, E. Liang, M. Elibol, Z. Yang, W. Paul, M. I. Jordan, and I. Stoica. Ray: A distributed framework for emerging ai applications, 2018. [28] G. I. Parisi, R. Kemker, J. L. Part, C. Kanan, and S. Wermter. Continual lifelong learning with neural networks: A review. Neural networks, 113:54–71, 2019. [29] J. Peleška and G. Šír. Redelex: A framework for relational deep learning exploration. In Machine Learning and Knowledge Discovery in Databases., pages 438–456. Springer Nature Switzerland, 2025. [30] J. Peleška and G. Šír. Tabular transformers meet relational databases. ACM Trans. Intell. Syst. Technol., 16(5), Sept. 2025. [31] J. Peleška and G. Šír. Task-Agnostic Contrastive Pretraining for Relational Deep Learning, June 2025. arXiv:2506.22530 [cs]. [32] R. Ranjan, V. Hudovernik, M. Znidar, C. Kanatsoulis, R. Upendra, M. Mohammadi, J. Meyer, T. Palczewski, C. Guestrin, and J. Leskovec. Relational Transformer: Toward Zero-Shot Foundation Models for Relational Data, Oct. 2025. arXiv:2510.06377 [cs]. [33] J. Robinson, R. Ranjan, W. Hu, K. Huang, J. Han, A. Dobles, M. Fey, J. E. Lenssen, Y. Yuan, Z. Zhang, X. He, and J. Leskovec. Relbench: A benchmark for deep learning on relational databases. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. [34] E. Rossi, B. Chamberlain, F. Frasca, D. Eynard, F. Monti, and M. Bronstein. Temporal graph networks for deep learning on dynamic graphs. arXiv preprint arXiv:2006.10637, 2020. [35] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio. Graph attention networks. In International Conference on Learning Representations, 2018. 11

[36] Y. Wang, X. Wang, Q. Gan, M. Wang, Q. Yang, D. Wipf, and M. Zhang. Griffin: Towards a Graph-Centric Relational Database Foundation Model, May 2025. arXiv:2505.05568 [cs]. [37] J. Wehrstein, C. Binnig, F. Ozcan, S. Vasudevan, Y. Gan, and Y. Wang. Towards foundation database models. In Proceedings of the 15th Annual Conference on Innovative Data Systems Research (CIDR), 2025. [38] J. White. PubMed 2.0. Medical Reference Services Quarterly, 39(4):382–387, Oct. 2020. [39] F. Wu, V. P. Dwivedi, and J. Leskovec. Large language models are good relational learners. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7835–7854, 2025. [40] J. Yin, P. Huo, B. Zhu, H. Yan, S. Wang, S. Pan, and C. Zhang. Rel-MOSS: Towards Imbalanced Relational Deep Learning on Relational Databases, Mar. 2026. arXiv:2603.07916 [cs]. [41] L. Zahradník, J. Neumann, and G. Šír. A deep learning blueprint for relational databases. In NeurIPS 2023 Second Table Representation Learning Workshop, 2023. [42] X. Zhang, D. Song, and D. Tao. Continual graph learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.

12

A

Technical appendices and supplementary material

This appendix provides additional implementation details, full data-distribution analyses, and extended experimental results that complement the main text. A.1

Detailed Experimental Setup and Compute Resources

We expand the experimental overview from Section 5.2 by summarizing the implementation stack and the main architectural, data-loading, and training choices. All models are implemented in PyTorch using PyTorch Lightning and PyTorch Geometric, and integrate with the RelBench [33] and REDELEX [29] libraries. Model Architecture and Initialization. As formalized in the source scripts, the core architecture relies on a HeterogeneousSAGE module. The relational databases are processed into temporal heterogeneous graphs following the primary key-foreign key (PK-FK) relationships. To initialize the node features from the raw database attributes, we utilize HeteroEncoder built upon PyTorch Frame [17], leveraging tabular architectures such as ResNet [14] to produce initial node embedding matrices of dimensionality 128. Crucially, temporal dynamics are injected via a HeteroTemporalEncoder. For a given prediction instance, this module computes relative time encodings based on the anchor time of the target entity (Section 3.3). These temporal embeddings are directly added to the initial node features before being passed to the GNN. The graph message passing is handled by a 2-layer HeteroGraphSAGE using sum aggregation. Finally, a single-layer MLP with batch normalization serves as the task-specific prediction head. All components are trained end-to-end to jointly learn the tabular representations and the graph structure. Graph Construction and Sampling. Given the large scale of the datasets, we utilize mini-batch training with a temporal neighbor sampling strategy. Specifically, we sample a maximum of 32 neighbors for the first GNN layer and 16 neighbors for the second layer (decaying by a factor of two per layer). To strictly prevent temporal leakage, we use a uniform temporal sampling strategy that only considers edges and nodes valid at or prior to the anchor timestamp (Section 3.3). For the Finetune (Upsampled) training regime (Section 4.2), we construct a specialized data loader that samples mini-batches from the newest data increment and the historical data with an equal 50% probability. Training and Optimization. The models are optimized using the Adam optimizer with an initial learning rate of 0.001. We use a uniform batch size of 128 for all tasks. During each episodic increment (Section 4), models are trained for a maximum of 2000 optimization steps. We artificially limit the training epochs to 100 batches per epoch to ensure frequent validation checks and early stopping. All experiments were orchestrated using Ray [27] to manage the distributed execution of the incremental training regimes (Section 4.2) across different time splits. Trials were automatically scheduled on available GPU resources, most commonly on the Tesla V100-SXM2-32GB. To ensure computational feasibility and simulate realistic production constraints, a strict maximum time limit of 2 hours was enforced for each individual incremental training step. Model checkpointing was handled by PyTorch Lightning, saving the best model weights based on the validation metric at each epoch. A.2

Data Analysis

Figures 6 and 7 present the full temporal distribution analysis for all 12 evaluated tasks. Whereas the main text focused on selected illustrative cases, these complete results confirm that the same qualitative patterns described there also hold more generally. In Figure 6, we see that tasks defined on a shared dataset exhibit similar distributions of samples across time windows (Section 4.1), including the categorization into the ‘linear’ and ‘exponential’ categories. In contrast, Figure 7 shows that the target value distributions vary considerably between tasks, with pronounced distributional shifts that are particularly evident for the tasks derived from the rel-f1 dataset. 13

Figure 6: Number of new samples from individual time windows ∆w and the cumulative size of the task over time.

14

Figure 7: Cumulative value of target y over the full range off the task windows ∆w. Mean value with standard deviation for regression tasks, positive class rate for the binary classification tasks.

A.3

Extended Results

This section reports the complete set of performance metrics for all evaluated tasks, extending the illustrative examples provided in the main text. Figure 8 presents the performance of the From Scratch baseline (Section 4.2) across all increments (Section 4). With the exception of rel-f1 driver-dnf and rel-trial site-success, all tasks display a clear performance improvement for episodes nearer to the validation horizon, underscoring the susceptibility of static models to temporal concept drift. Figures 9 and 10 contrast the different proposed incremental training regimes (Section 4.2). The results consistently indicate that our fine-tuning strategies substantially enhance performance in the few-shot scenario shown in Figure 10, highlighting the potential of knowledge transfer between 15

the episodes, and frequently also improve fully trained models, with the effect being especially pronounced for tasks defined on the rel-stack database. Tables 3 and 4 broaden the time-weighted evaluation from the main text (Section 4.3) by reporting standard deviations over 5 independent runs for regression and classification tasks, respectively. Notably, Table 4 highlights the consistently superior performance of the fine-tuning regimes, with the sole exception of rel-trial study-outcome, which remains challenging, as traditional tabular models can outperform RDL [33] on this task.

Figure 8: Performance of the From Scratch models trained in the multi-episodic setting, reported on the original validation and test splits. Metrics are Mean Absolute Error (MAE; lower indicates better performance) for regression tasks and AUROC (higher indicates better performance) for binary classification. 16

Figure 9: Top performance of different model training regimes in the multi-episodic setting, reporting the validation score for the subsequent increment. Metrics are Mean Absolute Error (MAE; lower indicates better performance) for regression tasks and AUROC (higher indicates better performance) for binary classification.

17

Figure 10: Performance of different model training regimes in the multi-episodic setting after first 100 training steps, reporting the validation score for the subsequent increment. Metrics are Mean Absolute Error (MAE; lower indicates better performance) for regression tasks and AUROC (higher indicates better performance) for binary classification.

18

Dataset

Training Regime

Task

Decay FT Cuml.

FT Incr.

FT Upspl.

Scratch

rel-avito

ad-ctr

0.0 0.3 0.9

0.037 ± 0.001 0.037 ± 0.001 0.035 ± 0.001

0.037 ± 0.001 0.037 ± 0.001 0.036 ± 0.001

0.039 ± 0.002 0.039 ± 0.002 0.037 ± 0.002

0.039 ± 0.002 0.038 ± 0.002 0.037 ± 0.002

rel-f1

driver-position

0.0 0.3 0.9

3.256 ± 0.056 2.858 ± 0.055 3.466 ± 0.135

3.408 ± 0.053 2.922 ± 0.021 3.293 ± 0.107

3.278 ± 0.104 3.058 ± 0.153 3.569 ± 0.316

3.106 ± 0.125 2.827 ± 0.116 3.358 ± 0.247

rel-stack

post-votes

0.0 0.3 0.9

0.061 ± 0.000 0.061 ± 0.001 0.059 ± 0.001

0.062 ± 0.001 0.062 ± 0.001 0.060 ± 0.001

0.062 ± 0.001 0.062 ± 0.001 0.060 ± 0.001

0.066 ± 0.002 0.065 ± 0.002 0.063 ± 0.002

0.0 0.3 0.9 0.0 0.3 0.9

0.428 ± 0.017 0.431 ± 0.016 0.440 ± 0.014 46.369 ± 0.104 46.565 ± 0.092 47.239 ± 0.120

0.395 ± 0.003 0.396 ± 0.003 0.399 ± 0.005 45.996 ± 0.321 46.163 ± 0.300 46.735 ± 0.267

0.419 ± 0.008 0.420 ± 0.006 0.421 ± 0.006 46.545 ± 0.304 46.819 ± 0.299 47.760 ± 0.327

0.393 ± 0.013 0.392 ± 0.011 0.389 ± 0.007 48.143 ± 0.183 48.281 ± 0.175 48.755 ± 0.170

site-success rel-trial study-adverse

Table 3: Performance of individual training regimes on regression tasks, trained on the complete training split, assessed using temporally decayed Mean Absolute Error with standard deviations (lower indicates better performance) over the combined original validation and test splits.

Dataset

Task

user-clicks rel-avito user-visits

driver-dnf rel-f1 driver-top3

user-badge rel-stack user-engagement

rel-trial

study-outcome

Training Regime

Decay FT Cuml.

FT Incr.

FT Upspl.

Scratch

0.0 0.3 0.9 0.0 0.3 0.9

0.659 ± 0.005 0.657 ± 0.004 0.645 ± 0.002 0.665 ± 0.003 0.667 ± 0.003 0.676 ± 0.003

0.659 ± 0.006 0.657 ± 0.006 0.646 ± 0.004 0.666 ± 0.002 0.669 ± 0.002 0.678 ± 0.002

0.656 ± 0.011 0.654 ± 0.010 0.641 ± 0.007 0.666 ± 0.001 0.669 ± 0.001 0.678 ± 0.001

0.648 ± 0.013 0.646 ± 0.012 0.637 ± 0.006 0.666 ± 0.003 0.668 ± 0.004 0.677 ± 0.004

0.0 0.3 0.9 0.0 0.3 0.9

0.763 ± 0.016 0.737 ± 0.034 0.695 ± 0.074 0.892 ± 0.006 0.779 ± 0.007 0.689 ± 0.018

0.818 ± 0.001 0.759 ± 0.015 0.716 ± 0.004 0.898 ± 0.003 0.779 ± 0.009 0.671 ± 0.016

0.776 ± 0.008 0.745 ± 0.024 0.687 ± 0.053 0.841 ± 0.009 0.725 ± 0.029 0.659 ± 0.053

0.814 ± 0.011 0.755 ± 0.016 0.670 ± 0.025 0.867 ± 0.016 0.754 ± 0.020 0.653 ± 0.048

0.0 0.3 0.9 0.0 0.3 0.9

0.889 ± 0.000 0.890 ± 0.000 0.894 ± 0.000 0.902 ± 0.001 0.902 ± 0.001 0.901 ± 0.001

0.890 ± 0.001 0.891 ± 0.001 0.894 ± 0.001 0.901 ± 0.001 0.901 ± 0.001 0.900 ± 0.001

0.889 ± 0.001 0.890 ± 0.001 0.893 ± 0.001 0.901 ± 0.001 0.901 ± 0.001 0.899 ± 0.001

0.882 ± 0.000 0.883 ± 0.001 0.887 ± 0.001 0.896 ± 0.001 0.896 ± 0.001 0.895 ± 0.002

0.0 0.3 0.9

0.633 ± 0.006 0.631 ± 0.006 0.626 ± 0.007

0.653 ± 0.005 0.651 ± 0.005 0.646 ± 0.004

0.638 ± 0.024 0.635 ± 0.024 0.626 ± 0.024

0.657 ± 0.004 0.656 ± 0.004 0.651 ± 0.005

Table 4: Performance of individual training regimes, trained on the complete training split, assessed using temporally decayed AUROC with standard deviations (higher indicates better performance) over the combined original validation and test splits.

19

Related documents

• Temporal Heterogeneous Graph Pretraining for Relational Deep Learning#1129816
• Reforge: Low-Latency Distributed GNN Serving with Selective Embedding Recomputation#1049613
• EdgeReMIND: A Scalable, Top-Ranked Memorization Baseline for Temporal Multi-Relational Link Prediction#1011630
• SkillNet: Create, Evaluate, and Connect AI Skills#242879
• Plain Transformers are Surprisingly Powerful Link Predictors#1118471