Understanding the Surprising Generalization Properties of Tabular Foundation Models

Understanding the Surprising Generalization Properties of Tabular Foundation Models Nour Shaheen1,2,3,† , Junwei Ma4,† , Alex Labach5 , Frank Hutter6,7,8 , Valentin Thomas9 , Anthony L. Caterini5

arXiv:2608.17957v1 [cs.LG] 18 Aug 2026

1

Polytechnique Montréal, 2 Mila – Quebec AI Institute, 3 Chandar Research Lab, 4 University of Toronto, 5 Layer 6 AI, 6 Prior Labs, 7 ELLIS Institute Tübingen, 8 University of Freiburg, 9 Cohere

Abstract Tabular Foundation Models (TFMs) increasingly rely on in-context learning, where a model receives labelled examples at inference time and predicts labels for new inputs without updating its weights. Existing TFMs are typically trained on either massive synthetic corpora or very large collections of real datasets. In contrast, we show that surprisingly strong transfer can emerge from self-supervised pre-training on just a single real table. In this setting, we also find that tables tend to be either broadly useful or broadly poor regardless of downstream prediction task, and that the strongest predictor of usefulness is the number of features rather than the number of instances. This leads to a task-centric interpretation of tabular pre-training: the number and the quality of tasks are essential for the pre-training of TFMs. We show that the same task-centric perspective can help corpus design at scale: fine-grained column-level pre-processing consistently improves downstream performance, while no improvements are observed when we filter or deduplicate at the dataset level. Finally, we offer a new perspective for how TFMs generalize: we believe that tabular in-context generalization is largely retrieval-based, and good models are those that learn to identify relevant examples in the provided context and aggregate them well. The mechanics of TFMs have been relatively understudied; our task-centric, retrieval-based perspective offers a new framework to guide future model and corpus design.

1

Introduction

Tabular data is commonly viewed as highly heterogeneous, leading to a prevailing belief that transfer learning across tabular domains is nearly impossible: e.g., how could training on handwritten digits possibly transfer to estimating Californian housing prices? However, recent tabular in-context learning (ICL) methods such as TabPFN [16, 17], TabDPT [22], and TabICL [27, 28]—termed tabular foundation models (TFMs)—have challenged this assumption. These methods train transformers using either massive amounts of synthetic data (millions of tables generated from structured priors [16, 17, 27, 28]) or by sampling from large collections of real tabular datasets [22, 12], randomizing context composition and prediction targets at each step to encourage in-context generalization. All modern tabular ICL models of which we are aware (e.g., [16, 17, 22, 27, 28, 34, 11, 31, 35]) share a similar task construction procedure during pre-training: a dataset is either selected or created, a column of this dataset is used as the target, and a subset of the remaining columns are used as features. Inference is then performed on new datasets while keeping the weights frozen: labelled examples from the evaluation dataset are given as context and the model predicts the values of unlabelled examples. The general belief is that, much like in large language models, only a very diverse pre-training set can cover the distribution of unseen tabular tasks and enable out-of-domain generalization [24, 16]. † Equal contribution.

Preprint.

MNIST

California Housing

Colleges 0.92

0.8

0.90

0

10000 20000 Gradient Steps

0.88

0.525 0.500

Colleges 0.70 0.65 0.60 0.55 0.50 0.025 0.000

CTR-23 Single Dataset Linear Random Forest XGBoost TabDPT

R2

R2

AUC

0.7 0.6 0.025 0.000

CC-18

0

10000 20000 Gradient Steps

0

10000 20000 Gradient Steps

Figure 1: Transfer from a single pre-training dataset. Left: Training only on vectorized MNIST (treated as a table) and evaluating on C ALIFORNIA H OUSING. Middle and Right: Training on the C OLLEGES dataset and evaluating on the full CC-18 and CTR-23 evaluation suites, respectively. Yet we find the opposite, and this surprising finding underpins our work: even with a single real-world table—such as vectorized MNIST [20]—a transformer trained with a naïve self-supervised learning (SSL) objective can still acquire generalization capabilities that transfer robustly across domains. Figure 1 illustrates this: a transformer trained from scratch only on the MNIST table yields strong in-context performance on structurally and semantically unrelated datasets like C ALIFORNIA H OUSING [26]. Similarly, training on the C OLLEGES dataset [14] and evaluating the resulting model on the OpenML CC-18 [3] and CTR-23 [9] benchmark suites (CC-BY license) ends up closely matching the performance of a random forest baseline. In this paper, we first investigate which properties of single real-world datasets are most linked to generalization. We find: (1) Datasets that transfer well tend to do so universally across evaluation sets; (2) Feature count is a strong predictor of generalization, while instance count matters little; and (3) The number of unique tasks a model is pre-trained on strongly drives generalization ability. We then apply this insight to pre-training on large data corpora. The task-centric view still applies with minor tweaks to control for data quality: more tasks generally help, as long as they have sufficient quality. In particular, removing duplicated, excessively correlated, or otherwise uninformative columns from each dataset further improves performance over baselines. Conversely, dataset-level filtering—even deduplication—is ineffective in our experiments. Although one might expect datasets resembling the “bad” single pre-training datasets to hurt performance, we find the opposite: adding them to “good” datasets still tends to improve generalization. Finally, we provide an explanation for what makes generalization possible for TFMs. We believe the original Bayesian interpretation [24, 16] is unsatisfying since it requires downstream tasks to be covered by the distribution of the prior, which directly contradicts single-table generalization. We instead argue that TFMs act as learned retrieval-and-aggregation procedures. We provide experimental evidence for this by showing (1) a strong link between a model’s performance and its ability to retrieve points within its own context, and (2) that architecture modifications enforcing nearest-neighbour-like behaviour have limited impacts on performance. We also show that the strongest single-dataset models display similar patterns in their attention maps, suggesting (although not proving) the existence of a universal retrieval space enabling generalization.

2

Generalization from single table training

We design experiments to explore the central question of what constitutes a “good dataset” such that a tabular ICL model pre-trained on it can generalize. To isolate the effects of the training datasets themselves, we fix both the model architecture and the pre-training procedure throughout most experiments. Specifically, we use the shared backbone architecture from TabPFNv1 [16] and TabDPT [22], a widely adopted design for tabular ICL. We use the pre-training procedure from TabDPT to train tabular ICL models from scratch using a single real dataset only; as explained above, this procedure is fundamentally similar to all standard TFM training protocols. To isolate the effect of data, we choose not to use retrieval mechanisms (as used in TabDPT) during pre-training in this section, although it is enabled during inference. We evaluate the generalization performance on established benchmarks in order to easily compare with reported baseline performances directly.

2

2.1

Are good pre-training datasets universally good?

In Figure 1, we showed the surprising finding that pre-training an ICL tabular model on MNIST can yield strong performance on unrelated datasets. We ask: “Is each pre-training dataset only good for some downstream task or does it generalize universally?” We pre-train models separately on each of Ntrain = 88 training datasets from the TabDPT pre-training data, spanning different modalities and sizes, and evaluate the resulting 88 models on Neval = 107 diverse datasets: the 72 CC-18 [3] classification datasets and 35 CTR-23 [9] regression datasets. No dataset appears in both pre-training and evaluation. Linear and RandomForest baselines were computed using scikit-learn defaults, except with logistic regression maximum iterations increased to 1000. XGBoost is the tuned baseline from McElfresh et al. [23], and TabDPT is computed from the public repository (Version 1.1, Apache-2.0 license).

(a) Ranking Correlation: The high values indicate that pre-training datasets which are good on one evaluation dataset tend to be good on others too.

(b) Domain to Domain Transfer: We do not observe stronger transfer when the pre-training and evaluation domain are shared.

Figure 2: Universality of dataset quality. (a) We compute the rank of training sets for each evaluation dataset and then plot a histogram of the Spearman correlation between the ranks on all pairs of evaluation datasets. Most evaluation datasets have very correlated ranks. (b) We group the training and evaluation datasets into distinct domains and plot average ranks (lower is better). Training and evaluation pairs from the same domain do not appear to transfer better than ones from different domains. In Figure 2a, for each evaluation dataset, we rank the pre-training datasets based on their performance on that task, leading to Neval rankings. We compute Spearman correlations over all ranking pairs, with higher values indicating stable relative rankings. Since the histogram is heavily left-skewed, it indicates that if a dataset is good for one evaluation dataset it tends to be good for others too, even across vastly different evaluation tasks. Next, we investigate whether a training dataset tends to generalize better to evaluation datasets within the same domain. We manually categorize both pre-training and evaluation datasets into domains, record the final performance (either accuracy for classification or correlation for regression) for each single training dataset, and then rank the training datasets from best to worst for each domain. Finally, we aggregate over the domains of the training datasets. If datasets from, e.g., Domain A are expected to transfer, on average, better to Domain A, we would expect a strong diagonal dominance in this matrix. However, in Figure 2b, we can see this is not the case: within-domain generalization is not higher. Rather, across the board, some domains are just broadly better for pre-training (e.g., other-science is great, deterministic-simulated is not). 2.2

What constitutes a good dataset?

To understand which properties of a training dataset yield successful generalization, we run a metaanalysis on the Ntrain pre-training datasets. For each dataset, we collect descriptive meta-features, including: the number of features, number of instances, number of categorical features, number of numeric features, amount of missing values, and final pre-training losses for both classification and regression heads. These meta-features are split into meta-train (80%) and meta-test (20%) sets. Using these meta-features, we train an XGBoost [6] regressor to predict a dataset’s average downstream generalization score, defined as the average of correlation for regression tasks and accuracy for classification tasks. The resulting model achieves an R2 of 0.67, indicating that roughly 3

two-thirds of the variance in generalization can be reproduced from straightforward dataset properties alone. Notably, when evaluating feature importance (see Figure 3a), the number of features emerges as by far the strongest predictor of downstream transfer performance. In contrast, the number of instances—often presumed critical in classical settings [33]—shows negligible predictive power. This suggests that the richness of the feature space is far more important than purely the dataset size for tabular ICL-based generalization. We validate this in Figure 3b by removing a fraction of either rows or columns on the C OLLEGES dataset, showing that features tend to matter much more than instances. After dropping about 70% of features, the model performance drops to a similar level as a linear model, while dropping more than 70% of instances still allows the model to maintain performance close to a random forest. We see a similar phenomenon across many other datasets. XGBoost Feature Importances

n_features 0.080 train_loss_cls 0.064 train_loss_reg n_numeric 0.045 n_missing 0.016 n_instances 0.012 n_categorical 0.012 0.0

0.1

Effect of Dropping Feature or Instance

0.771

AUC (CC-18)

0.92 0.91

Varying Features Varying Instances Linear (baseline) Random Forest (baseline) XGBoost (baseline) TabDPT (baseline)

0.90 0.89

0.2

0.3

0.4

0.5

0.6

Feature Importance

0.7

0.8

0.0

(a) Feature importance from an XGBoost model trained to predict average performance across 107 downstream tasks from dataset metadata.

0.1

0.2

0.3

0.4

% Dropped

0.5

0.6

0.7

(b) Column vs. Row Importance: We analyze the downstream performance when removing either features or instances from the COLLEGES dataset.

Figure 3: Not all cells in a table are made equal: the number of features matters much more than the number of instances as pre-training dataset for tabular ICL models. Training on many good tasks unlocks generalization

To understand why dropping features caused a strong performance drop, we revisit the randomized procedure used for pre-training: a column is selected as target while a subset of other columns are used as features; we call this a task. Using a fixed supervised target provides just one task, while using a randomized target and features can have O(k 2k ) tasks for k columns. Thus, the role of columns may be more important than the role of rows as it is directly tied to the number of tasks or feature relationships the model is exposed to during training.

0.9

AUC on CC-18

2.3

0.8 0.7 0.6 0.5

5

50

500

5,000

Number of Tasks

50,000

ipums_la_99-small ipums_la_99-small (max) musk musk (max) philippine philippine (max) satellite satellite (max) twonorm twonorm (max)

Figure 4: Downstream AUC vs. the number of unique tasks used during training. (max) denotes the unrestricted setting where the sampler draws from the full set of eligible task configurations with no upper count limit. More tasks during pre-training consistently leads to better transfer.

We disentangle the impact of the number of tasks from the number of features in Figure 4. We do not change the amount of instances or features in a given dataset but instead we vary the number of tasks the model is allowed to pre-train on. To be precise, we select N tasks using allowed permutations (one target, the remaining columns being either masked or used as features) and vary N for several datasets. A striking pattern emerges: across all datasets, increasing the number of tasks from 5 to tens of thousands drives the performance from barely above random (AUC = 0.5) to good. Thus, although inherent dataset differences matter and some datasets are significantly better for pre-training, the main factor explaining cross-domain generalization is the number of tasks seen during pre-training. In other words, the key is not the amount of pre-training data in tokens or bytes of memory, but the number of tasks/feature-target relationships that enable a tabular ICL model to generalize. We further support the task-diversity view by training models on pairs with one dataset from the top quartile and one from the bottom quartile of Table 3. Pairing a strong dataset with a weak one does not hurt performance and can even slightly improve it: C OLLEGES (AUC = 0.918 alone) with A NALCATDATA _S UPREME (0.814 alone) reaches 0.919, and APSFAILURE (0.918 alone) with P OKER -H AND (0.777 alone) also reaches 0.919. These results suggest that datasets performing poorly during single-table pre-training are not intrinsically uninformative: they may be task-poor, but their tasks can complement those of a stronger dataset. This pattern also reappears in the next section. 4

3

From one to many: pre-training insights for large corpora

The preceding section showed how pre-training data properties such as feature count, sample size, and task diversity affect generalization in the single-dataset regime. We extend these observations to the practical setting where a TFM is pre-trained on a large, heterogeneous corpus of real-world tabular datasets. We ask the question of how to curate and pre-process such a corpus. We ablate along two axes: (i) dataset-level selection: which datasets to include in the corpus, and (ii) feature-level pre-processing: how to filter the features of each dataset prior to training. Our key findings are that exact deduplication is unnecessary, aggressive feature-based dataset-level filtering hurts performance, and lightweight column-level pre-processing is consistently beneficial. We use part of the datasets collected by Hosseinzadeh et al. [18] as our pre-training corpus: 1,732 datasets collected from OpenML [4], excluding datasets from the CC-18, CTR-23, and TabArena benchmarks [8], which we use for evaluation. We use a training context length of 1,024 (512 queries) and an inference context length of 2,048, and do not pre-process evaluation datasets. Our main evaluation metric is Interquartile Mean (IQM), which is standard and robust to outliers [1, 22]. We defer the full experimental setup, including model and training details, to Appendix B.1. 3.1

Does dataset-level selection help?

Two natural corpus-curating strategies would be deduplication and filtering out “universally bad” datasets. We find that neither is beneficial. Deduplication: Although deduplication benefits LLM training [21], exact deduplication is unnecessary here: models trained on the full corpus (1,732 datasets) and a deduplicated corpus (1,535 datasets, identified by the SHA-256 hash of X and y) are statistically indistinguishable on CC-18 and CTR-23 (Table 1). We attribute this to the SSL objective, which already reduces the impact of duplicates through random column-target sampling and independent context windows at each step. Dataset-level filtering: A reductive direction here would be to restrict the pre-training corpus to feature-rich datasets, since we established that those generalize better than low-feature ones. However, we previously showed that even a table with very few features can induce non-trivial out-of-domain generalization. Low-feature datasets are qualitatively different from high-feature ones, covering simpler dependency structures, smaller column-count regimes, and distinct domains. We test this by restricting both the full and deduplicated corpus to datasets with at least 30 features (607 datasets, and 557 after deduplication). Table 1 shows that this degrades performance across the board, with similar performance regardless of deduplication: excluding low-feature datasets reduces the number of diverse tasks in a way not offset by higher average per-dataset quality. 3.2

Column-level pre-processing as a better alternative

Since coarse dataset filtering is not fruitful, we instead perform fine-grained feature cleaning to increase task quality and diversity. For each dataset, we apply a subset of three operations, each ablated individually by training a separate model for every combination. NaN-column drop removes columns where more than 50% of values are missing; after TabDPT’s mean imputation, such columns become nearly constant, so dropping them improves task quality. Correlation deduplication greedily removes columns until no pair has absolute correlation above 0.90, improving task diversity. The target y is included in the joint correlation matrix so features near-collinear with it are also removed, while y itself is protected from removal. We test Pearson and Spearman variants. At each greedy step, the column with the greatest number of above-threshold correlation partners is removed, with ties broken by lower variance. All columns are treated as numeric: categorical features are pre-encoded as integers and used as-is. Minimum-feature enforcement (mf-k) skips any dataset for which fewer than k feature columns remain after the above steps; we set k = 5 by default. Unlike the two column operations above, mf-k is a dataset-level exclusion rule rather than a column transformation: it is applied after column cleaning to drop tables left with too few usable columns. It’s a light cleaning step that preserves most of the corpus rather than an aggressive dataset selection rule that discards whole task pools. Table 1 reports the full ablation. Every pre-processing variant outperforms the baseline on regression, with IQM R2 gains from +0.002 to +0.014. Classification gains are smaller but broadly consistent: all variants improve accuracy, and most improve AUC. This is consistent with the single-dataset findings: generalization is governed by the number of unique, good tasks the model is pre-trained on. Dataset filtering discards entire task pools, whereas column pre-processing removes only redundant or uninformative tasks without shrinking the corpus, thus increasing task diversity and quality. 5

Table 1: Dataset-level selection and column-level pre-processing ablations. Results are IQM with 95% CI; AUC/accuracy on CC-18, correlation/R2 on CTR-23, Avg. is unweighted. First row is the full-corpus baseline; dataset-level rows test deduplication, ≥ 30 feature filtering, and both. Column-level Setup = (NaN, Corr, mf-5): ‘+’ enabled, ‘–’ disabled; NaN drops columns with > 50% missing, Corr is Pearson (P) or Spearman (S) deduplication at threshold 0.90, and mf-5 skips datasets with < 5 surviving features. Best column-level IQM per metric is bold. Setup Baseline

AUC

Acc.

Corr.

R2

Avg.

0.9757[.9731,.9776] 0.9233[.9199,.9265] 0.9021[.8993,.9047] 0.8145[.8097,.8188] 0.9039

Dataset-level selection Deduped 0.9752[.9728,.9773] 0.9236[.9202,.9269] 0.9019[.8989,.9047] 0.8146[.8101,.8189] 0.9038 ≥ 30 0.9728[.9704,.9748] 0.9208[.9175,.9241] 0.8882[.8832,.8930] 0.7890[.7812,.7966] 0.8927 ≥ 30 + dedup 0.9724[.9701,.9744] 0.9195[.9159,.9231] 0.8871[.8817,.8921] 0.7870[.7788,.7949] 0.8915 Column-level pre-processing (+, –, –) 0.9760[.9734,.9779] 0.9256[.9226,.9286] 0.9078[.9049,.9105] 0.8221[.8177,.8261] 0.9079 (–, P, –) 0.9760[.9737,.9780] 0.9251[.9222,.9280] 0.9043[.9019,.9065] 0.8195[.8157,.8230] 0.9062 (–, S, –) 0.9758[.9734,.9778] 0.9238[.9209,.9267] 0.9032[.9006,.9055] 0.8164[.8124,.8201] 0.9048 (+, –, +) 0.9763[.9737,.9783] 0.9251[.9221,.9280] 0.9079[.9053,.9103] 0.8249[.8207,.8289] 0.9085 (+, P, –) 0.9756[.9732,.9777] 0.9239[.9204,.9272] 0.9027[.9002,.9050] 0.8166[.8121,.8205] 0.9047 (–, P, +) 0.9759[.9734,.9778] 0.9250[.9220,.9280] 0.9098[.9071,.9125] 0.8281[.8236,.8323] 0.9097 (+, P, +) 0.9756[.9731,.9775] 0.9248[.9219,.9277] 0.9060[.9034,.9083] 0.8227[.8186,.8264] 0.9073 (+, S, –) 0.9758[.9734,.9778] 0.9241[.9215,.9267] 0.9057[.9031,.9082] 0.8203[.8161,.8241] 0.9065 (–, S, +) 0.9762[.9737,.9782] 0.9255[.9228,.9282] 0.9078[.9053,.9102] 0.8241[.8203,.8278] 0.9084 (+, S, +) 0.9763[.9738,.9784] 0.9272[.9241,.9302] 0.9077[.9055,.9097] 0.8251[.8213,.8284] 0.9091

3.3

Does this extend to larger models, context windows, and other evaluation benchmarks?

We next ask whether the gains from column-level pre-processing generalize beyond the base model size, training context, and evaluation setting. To this end, we scale up along two axes simultaneously: model capacity (depth and width) and training context length. We train a deeper, wider model under three maximum training context lengths: 1,024 (“1k”, base), 2,048 (“2k”), and 4,096 (“4k”) tokens. Each variant is trained with and without pre-processing (we choose NaN + Spearman + mf-5, denoted “+pp”). Inference continues to use context size 2,048 and no pre-processing for consistency. Full architectural, training, and context size details for all six variants are listed in Appendix B.2.

Figure 5: Pre-processing yields Elo gains at all context sizes. Purple bars are models trained with pre-processing (+pp), blue bars their corresponding baselines, and grey bars other TabArena methods. TabDPT [22] is in orange. We evaluate on TabArena’s 51 tabular prediction tasks (Apache License 2.0) [8]. We compute pairwise win rates across all configurations. The results are directionally consistent with our findings: the pre-processed variant outperforms its baseline counterpart at every context scale: 4k+pp beats 4k on 67% of tasks, 2k+pp beats 2k on 61%, and 1k+pp beats 1k on 63%. The strongest overall configuration is 4k+pp. We contextualize these findings within the broader TabArena leaderboard in Figure 5, reporting Elo against some other default-configuration baselines, with details in Appendix C.

6

4

Understanding what makes generalization possible

The previous sections established two key empirical facts: first, that broad transfer can emerge from pre-training on a single table, and second, that this behaviour persists and can be systematically shaped when scaling to large corpora. These observations cannot be easily explained as parametric transfer learning where predictive signals from one dataset transfer to a related one. Instead, the model must have learned a general prediction algorithm, and we argue that tabular ICL models specifically rely on a learned retrieval-and-aggregation procedure. In that sense, their form of generalization is closer to that of k-nearest neighbours than a standard fitted predictor—indeed, it would not be surprising for a non-parametric method such as kNN to perform reasonably well on both an MNIST-like task and a C ALIFORNIA -H OUSING-like task. The role of pre-training is thus not to store a universal decision rule in the weights, but rather to learn how to compare examples, identify context points relevant to a query, and aggregate their label information. This retrieval-based perspective stands in contrast to previous works on prior-fitted networks, which interpret TFM pre-training as inducing a prior that corresponds to the space of processes that generate pre-training datasets [16, 24]. In Section 2, however, we observed that TFMs are effective on data outside of their training distribution, transferring between entirely different tabular datasets, and furthermore are effective when pre-trained with a very limited distribution. The prior-fitting account still does not explain the case where pre-training and inference distributions have no overlap, which motivates our alternative retrieval-and-aggregation perspective of model behaviour. Concurrent work casts further doubt on the prior-fitting account by showing that the predictions of current TFMs are neither marginalization- nor factorization-consistent, and therefore cannot be the conditionals of any single joint distribution over a table’s columns [19]. We concede that approximate Bayesian inference could be occurring in models trained with broader distributions, but we believe that learned retrieval is a sufficient explanation for these models’ capabilities, while being both more parsimonious and consistent with our observations. In this section, we provide evidence for learned retrieval through three controlled analyses. First, we show that models that generalize well are also better at retrieving exact label information from the context. Second, we show that forcing the model to behave in a more explicit similarity-based way does not hurt performance, and can even help in weaker regimes. Finally, we show that strong models tend to converge toward similar retrieval patterns, suggesting that there exists a fairly universal structure underlying good tabular generalization. 4.1

Generalization is tied to the ability to retrieve from the context

Based on our interpretation, the ability to generalize to new datasets should be tightly linked to a more primitive capability: the ability to identify and use the relevant information already present in the context. A model that cannot reliably retrieve useful in-context examples would have little hope of producing robust predictions on novel datasets. Conversely, a model that does generalize well should also perform well on tasks where the solution is effectively already available in the context and the main challenge is to recover it. To test this, we construct a controlled retrieval task. For each model pre-trained on a single table, for evaluation, we run standard ICL inference, but with the query rows set to be the same as context rows, Figure 6: In-context retrieval AUC effectively testing whether the model can correctly retrieve (x-axis) vs. base AUC (y-axis) for and match identical inputs. Concretely, we use the CC-18 88 models each pre-trained on one classification datasets with a stratified subsample of up to 1024 dataset. Each point is one model. examples. The model is then given a sequence consisting of labelled examples followed by the same examples without labels. Performance is evaluated using AUC against the true labels, which we call retrieval AUC. Because every query row has an exact copy with its label in the context, a model that reliably attends to and copies the matching support label should score highly, although not perfectly: aggregation over several context rows can dilute or override the exact match, exactly as in kNN with k > 1. A model that ignores the context and relies purely on parametric knowledge would revert to marginal

7

or random predictions. The procedure is repeated with n = 3 independent stratified subsamples per dataset and the scores are averaged. We report the mean score across CC-18 datasets. The reported base AUC for each model is its held-out test-set AUC on CC-18 (standard ICL with distinct query and context rows) which we use as a proxy for model performance. This highlights the retrieve and copy skill as a core mechanism that tabular ICL must learn in order to generalize, similarly to the induction head mechanism in LLMs [25]. Results Figure 6 shows results for 88 single-table pre-trained models. We find a strong correlation (Pearson = 0.89) between the standard generalization performance on held-out subsets and models’ performance in this retrieval-oriented setting. Models that are weak at out-of-distribution generalization are also weak at exploiting the labelled context itself, whereas models that generalize well are precisely those that are able to recover the relevant support information. This supports the view that tabular ICL is not primarily driven by memorizing a broad family of parametric predictors in the weights. Instead, a substantial part of the problem appears to consist of learning a robust similarity-based procedure for retrieving and aggregating information from the context. We stress that exact-copy retrieval and downstream performance are distinct capabilities, measured under different prediction protocols, so the two AUCs are not comparable in absolute terms. The informative signal is how they vary together. 4.2

Forcing a retrieval-based attention mechanism improves performance for weaker models

If good generalization relies on learned retrieval, then one should expect tabular ICL models to be relatively robust to interventions that make them behave more explicitly like nearest-neighbour methods. Specifically, if such models identify similar context points and aggregate their labels, then constraining the attention mechanism to a simpler similarity rule should not degrade performance, but actually improve it in regimes where the model fails to generalize, by providing a useful inductive structure. We evaluate this by sharing the query and key projection weights for each attention layer, thus making each layer more “kNN-like” as the attention similarity score explicitly encodes a weighted inner product. Indeed, as we use qk-normalization, the attention score between points xi and xj when setting WQ = WK becomes qi⊤ qj = constant − 12 ∥qi − qj ∥2 . This intervention strictly reduces the expressiveness of the attention mechanism—making it closer to a Gaussian kNN—and tests whether the model would only require a symmetric comparison operator in order to generalize. pollen (OpenML 871)

puma8NH (OpenML 816)

0.75

0.700 0.675

0.70

0.70 0.65 normal/default q=k tied

0.60

Validation AUC

Validation AUC

poker-hand (OpenML 1567)

0.75

0.650

0.65

0.625 0.600

0.60 normal/default q=k tied

0.55

0.5

Validation AUC

0.80

0.575

0.5 0

512

1024

Gradient steps

1536

2048

normal/default q=k tied

0.550 0.5

0

512

1024

Gradient steps

1536

2048

0

512

1024

Gradient steps

1536

2048

(a) Tables that generalize poorly: pollen, puma8NH, poker-hand (left to right). colleges (OpenML 42727) 0.86

0.84

0.84

0.82

0.82

0.80 0.78 0.76 normal/default q=k tied

0.74 0.72 0.5

0.84

0.80 0.78 0.76 normal/default q=k tied

0.74 0.72 0.5

0

512

1024

Gradient steps

1536

2048

Census-Income (OpenML 4535)

0.86

Validation AUC

0.86

Validation AUC

Validation AUC

APSFailure (OpenML 41138)

0.82 0.80 0.78 normal/default q=k tied

0.76 0.5

0

512

1024

Gradient steps

1536

2048

0

512

1024

Gradient steps

1536

2048

(b) Tables that already generalize well: APSFailure, Colleges, Census-Income (left to right).

Figure 7: Ablation forcing WQ = WK on six single-table pre-training corpora, grouped by how well the default architecture generalizes with single-table pre-training.

8

Validation AUC

Raw Corpus: CC-18 AUC We first target pre-training tables that generalize poorly, 0.9325 i.e., tables whose resulting models attain low downstream 0.9300 AUC across the evaluation suite (selected using Table 3). 0.9275 This is where (i) the default architecture struggles to exploit labelled context examples from the pre-training table alone, 0.9250 and (ii) where the WQ = WK constraint should help 0.9225 most under the retrieval interpretation. In Figure 7a, the 0.9200 simplified version wins on all three such tables (pollen, 0.9175 puma8NH, poker-hand), by +0.1088 AUC on average, Default Q=K 0.9150 with analogous results on two further weak tables in Appendix E. On three tables that already generalize well 0.5 (APSFailure, Colleges, Census-Income), where an 0 20000 40000 60000 80000 100000 120000 Gradient steps imposed similarity rule has little left to contribute, the constraint gives no systematic gain but also no substantial Figure 8: Mean CC-18 AUC for the dedegradation, with a mean difference of −0.0027 AUC fault architecture and the WQ = WK (Figure 7b). Sharing the query and key projections variant, both pre-trained on the full therefore supplies a retrieval-inducing bias that matters OpenML corpus of Section 3. when the pre-training table is too weak to teach a stable comparison rule, and becomes redundant, or at most mildly restrictive, once it is not. Finally, we scale the intervention to the full OpenML pre-training corpus used in Section 3, where the two architectures are effectively tied: the constraint changes AUC by +0.0001 (Figure 8). We do not claim that WQ = WK is a better architecture, as this gap is negligible. The relevant point is that reducing attention to a strictly symmetric similarity rule has no noticeable cost even given abundant and diverse pre-training data, given the model sizes we used in our experiments. Along with the weak-table gains, this reinforces the interpretation of tabular ICL as mostly retrieval-based in this setting, since distances between tokens under WQ = WK are symmetric and analogous to soft-kNN.

4.3

Strong models converge toward similar retrieval patterns We further show that successful models should not merely perform well, but should do so in similar ways. If there exists a fairly universal retrieval strategy that works across tabular problems, then independently trained strong models should tend to converge toward comparable attention patterns. By contrast, if performance arose from brittle or highly idiosyncratic solutions, one would expect much more diversity across models.

We test this prediction directly on 52 TabDPT checkpoints each trained on a single OpenML dataset (single attention Figure 9: Pair attention similarity vs. head per block for simplicity, model and training details mean AUC, n = 2652 points (one per are in Appendix D). We probe each model on two further model pair × probe dataset), averaged held-out OpenML datasets via forward passes to extract over blocks. Spearman correlation is attention maps. For each (model, probe), we record 0.537. Per-block results are in Figure 11. the per-block, per-query attention distribution at a fixed in-context cutoff: the softmax-normalized vector a query token assigns over the C context tokens. We compare two models row-by-row using Spearman rank correlation, averaging over queries and probe examples to obtain one similarity scalar per (model pair, block, probe) and then average the resulting scalar over the 12 blocks, yielding one pair-level value per (model pair, probe). Across them, the Spearman correlation between mean held-out AUC and pair attention similarity is 0.537 (Figure 9). To isolate what drives this correlation, Table 2: Pair attention similarity by category. CIs are nonwe group models by performance, split- parametric bootstrap intervals (10,000 resamples) on the ting by AUC quartile. We form three difference of group means. pair categories: strong–strong (SS, both Group Mean n vs. ∆ 95% CI endpoints in the top quartile), strong– weak (SW), and weak–weak (WW, both SS 0.504 156 SW +0.163 [+0.143, +0.183] SW 0.341 338 WW +0.051 [+0.027, +0.075] in the bottom quartile). The results of WW 0.290 156 this experiment are in Table 2 and the

9

full methodological details (model specification, similarity definition, bootstrap-CI procedure, etc.) are in Appendix D, which also contains per-block correlation values. This convergence is unlikely to be accidental. Although checkpoints are trained on different datasets, high-AUC models independently discover similar attention patterns, as expected if good generalization follows a common retrieval principle. Together with the preceding analyses, these results suggest that tabular ICL models generalize not because they encode one gigantic universal predictor in their weights, but because they learn how to use the context at inference time. A simple similarity rule is often already enough to obtain non-trivial transfer, which explains why generalization is possible at all. The strongest models then improve upon this baseline by learning a more refined, more robust notion of relevance and aggregation.

5

Related Work

Although deep learning and gradient boosted tree ensemble algorithms are currently dominant on tabular benchmarks [8], our work demonstrates a connection to k-nearest neighbours [10] and possibly kernel-based extensions [36]. Explicit k-nearest retrieval has had recent applications as part of tabular ICL models [32, 22, 34]. While previous works on tabular ICL have explored many aspects of training task creation, they have generally focused on scaling the diversity of tasks, either through complex synthetic priors [16, 17, 27, 28, 34, 35] or augmented real world data [22, 12], and the regime of extremely limited pre-training data has not been explored. Other surprising generalization capabilities of tabular ICL models under modified training approaches include resilience to distribution shifts [15] and causal inference [2, 30]. Substantial literature exists analyzing and extending in-context learning in large language models [7], however this is an emergent capability seen in autoregressive next-token prediction models trained on large amounts of text data, which is a fundamentally different setting from TFMs that directly train on an ICL objective. The observations in this paper are therefore separate from LLM ICL behaviour. While some LLM-focused papers have evaluated non-text data (e.g., [5, 29, 13]), they have still focused on autoregressive sequence modelling in ways that make them inapplicable to the models discussed here. Tabula-8B [11] is an exception among tabular ICL models in that it uses a language model backbone, so we exclude it from our scope, but LLM-focused ICL findings could apply to it.

6

Conclusion and limitations

In this paper we investigated how pre-training data enables out-of-distribution generalization in TFMs. Taking a task-centric view, we explained why strong generalization can emerge even when training on a single real table, and expanded on this finding to make practical recommendations regarding datasetlevel and column-level data filtering. Finally, we argued for a retrieval-focused interpretation of TFMs rather than one based on standard transfer learning or Bayesian inference, providing empirical evidence for retrieval behaviours. Overall, our findings challenge existing assumptions about how TFMs work and what actually matters when scaling TFM pre-training data. Although we analyzed TFM behaviour across a broad range of datasets, model sizes, and context lengths, we acknowledge other limitations in our work. We were unable to definitively identify or exclude additional mechanisms beyond learned retrieval that may underlie out-of-distribution generalization. Further research is therefore required to achieve a more comprehensive understanding of how in-context learning operates in TFMs. Moreover, our investigation was restricted to pre-training data derived from real-world datasets, and consequently, our findings may not fully extend to scenarios in which models are trained exclusively on synthetic data. Finally, while transformer architectures are universal across modern TFMs, they are split between row-based and cell-based attention, and our results only used row-based attention following TabDPT and TabPFNv1.

7

Acknowledgements

This work was supported by Mitacs through the Mitacs Accelerate program, and was enabled in part by compute resources provided by Mila (mila.quebec) and the Digital Research Alliance of Canada. We also thank Rasa Hosseinzadeh for the valuable discussions and ideas that contributed to this work. 10

References [1] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In Advances in Neural Information Processing Systems, 2021. [2] Vahid Balazadeh Meresht, Hamidreza Kamkari, Valentin Thomas, Junwei Ma, Bingru Li, Jesse Cresswell, and Rahul Krishnan. CausalPFN: Amortized causal effect estimation via in-context learning. In Advances in Neural Information Processing Systems, 2025. [3] Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer, Pieter Gijsbers, Frank Hutter, Michel Lang, Rafael Gomes Mantovani, Jan van Rijn, and Joaquin Vanschoren. OpenML benchmarking suites. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021. [4] Bernd Bischl, Giuseppe Casalicchio, Taniya Das, Matthias Feurer, Sebastian Fischer, Pieter Gijsbers, Subhaditya Mukherjee, Andreas C Müller, László Németh, Luis Oala, Lennart Purucker, Sahithya Ravi, Jan N van Rijn, Prabhant Singh, Joaquin Vanschoren, Jos van der Velde, and Marcel Wever. OpenML: Insights from 10 years and more than a thousand papers. Patterns, 6 (7), 2025. [5] Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. Data distributional properties drive emergent in-context learning in transformers. In Advances in Neural Information Processing Systems, 2022. [6] Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SigKDD International Conference on Knowledge Discovery and Data Mining, 2016. [7] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, et al. A survey on in-context learning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1107–1128, 2024. [8] Nick Erickson, Lennart Purucker, Andrej Tschalzev, David Holzmüller, Prateek Desai, David Salinas, and Frank Hutter. TabArena: A living benchmark for machine learning on tabular data. In Advances in Neural Information Processing Systems, 2025. [9] Sebastian Felix Fischer, Matthias Feurer, and Bernd Bischl. OpenML-CTR23 – A curated tabular regression benchmarking suite. In AutoML Conference (Workshop), 2023. [10] Evelyn Fix and J. L. Hodges. Discriminatory Analysis: Nonparametric Discrimination, Consistency Properties, volume 1. USAF School of Aviation Medicine, 1951. [11] Josh Gardner, Juan C Perdomo, and Ludwig Schmidt. Large scale transfer learning for tabular data via language modeling. In Advances in Neural Information Processing Systems, 2024. [12] Anurag Garg, Muhammad Ali, Noah Hollmann, Lennart Purucker, Samuel Müller, and Frank Hutter. Real-TabPFN: Improving tabular foundation models via continued pre-training with real-world data. arXiv:2507.03971, 2025. [13] Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn in-context? A case study of simple function classes. In Advances in Neural Information Processing Systems, 2022. [14] Pieter Gijsbers. Colleges. OpenML Dataset 42727, 2020. URL https://www.openml.org/ d/42727. Accessed: 2025-10-21. [15] Kai Helli, David Schnurr, Noah Hollmann, Samuel Müller, and Frank Hutter. Drift-resilient TabPFN: In-context learning temporal distribution shifts on tabular data. In Advances in Neural Information Processing Systems, 2024. [16] Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. TabPFN: A transformer that solves small tabular classification problems in a second. In International Conference on Learning Representations, 2023. 11

[17] Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirrmeister, and Frank Hutter. Accurate predictions on small data with a tabular foundation model. Nature, 637(8045):319–326, 2025. [18] Rasa Hosseinzadeh, Alex Labach, Zexin Xue, Shuyi Han, Valentin Thomas, and Anthony L Caterini. TabDPT-Turbo: Efficient in-context learning for tabular prediction. arXiv:2608.01400, 2026. [19] Christian Klötergens, Vijaya Krishna Yalavarthi, Lars Schmidt-Thieme, and Tom Hanika. Do tabular foundation models agree with themselves? arXiv:2608.06004, 2026. [20] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. [21] Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8424–8445, 2022. [22] Junwei Ma, Valentin Thomas, Rasa Hosseinzadeh, Hamidreza Kamkari, Alex Labach, Jesse C. Cresswell, Keyvan Golestan, Guangwei Yu, Anthony L. Caterini, and Maksims Volkovs. TabDPT: Scaling tabular foundation models on real data. In Advances in Neural Information Processing Systems, 2025. [23] Duncan McElfresh, Sujay Khandagale, Jonathan Valverde, Vishak Prasad C, Ganesh Ramakrishnan, Micah Goldblum, and Colin White. When do neural nets outperform boosted trees on tabular data? In Advances in Neural Information Processing Systems, 2023. [24] Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. Transformers can do Bayesian inference. In International Conference on Learning Representations, 2022. [25] Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. arXiv:2209.11895, 2022. [26] R. Kelley Pace and Ronald Barry. Sparse spatial autoregressions. Statistics & Probability Letters, 33(3):291–297, 1997. [27] Jingang Qu, David Holzmüller, Gaël Varoquaux, and Marine Le Morvan. TabICL: A tabular foundation model for in-context learning on large data. In International Conference on Machine Learning, 2025. [28] Jingang Qu, David Holzmüller, Gaël Varoquaux, and Marine Le Morvan. TabICLv2: A better, faster, scalable, and open tabular foundation model. In International Conference on Machine Learning, 2026. [29] Gautam Reddy. The mechanistic basis of data dependence and abrupt learning in an in-context classification task. arXiv:2312.03002, 2023. [30] Jake Robertson, Arik Reuter, Siyuan Guo, Noah Hollmann, Frank Hutter, and Bernhard Schölkopf. Do-PFN: In-context learning for causal effect estimation. In Advances in Neural Information Processing Systems, 2025. [31] Marco Spinaci, Marek Polewczyk, Maximilian Schambach, and Sam Thelin. ConTextTab: A semantics-aware tabular in-context learner. In Advances in Neural Information Processing Systems, 2025. [32] Valentin Thomas, Junwei Ma, Rasa Hosseinzadeh, Keyvan Golestan, Guangwei Yu, Maksims Volkovs, and Anthony L Caterini. Retrieval & fine-tuning for in-context tabular models. Advances in Neural Information Processing Systems, 2024. 12

[33] Vladimir Vapnik. The Nature of Statistical Learning Theory. Springer Science & Business Media, 2013. [34] Xingxuan Zhang, Gang Ren, Han Yu, Hao Yuan, Hui Wang, Jiansheng Li, Jiayun Wu, Lang Mo, Li Mao, Mingchao Hao, Ningbo Dai, Renzhe Xu, Shuyang Li, Tianyang Zhang, Yue He, Yuanrui Wang, Yunjia Zhang, Zijing Xu, Dongzhe Li, Fang Gao, Hao Zou, Jiandong Liu, Jiashuo Liu, Jiawei Xu, Kaijie Cheng, Kehan Li, Linjun Zhou, Qing Li, Shaohua Fan, Xiaoyu Lin, Xinyan Han, Xuanyue Li, Yan Lu, Yuan Xue, Yuanyuan Jiang, Zimu Wang, Zhenlei Wang, and Peng Cui. LimiX: Unleashing structured-data modeling capability for generalist intelligence. arXiv:2509.03505, 2025. [35] Xiyuan Zhang, Danielle Maddix Robinson, Junming Yin, Nick Erickson, Abdul Fatir Ansari, Boran Han, Shuai Zhang, Leman Akoglu, Christos Faloutsos, Michael Mahoney, Tony Hu, Huzefa Rangwala, George Karypis, and Yuyang (Bernie) Wang. Mitra: Mixed synthetic priors for enhancing tabular foundation models. In Advances in Neural Information Processing Systems, 2025. [36] Wangmeng Zuo, David Zhang, and Kuanquan Wang. On kernel difference-weighted k-nearest neighbor classification. Pattern Analysis and Applications, 11(3):247–257, 2008.

13

A

Impact of pre-training dataset on CC-18 and CTR-23 Performance

Tables 3 and 4 report the performance of TabDPT when pre-trained on different datasets and evaluated on two benchmark suites: CC-18 for classification (measured by AUC) and CTR-23 for regression (measured by R2 ). Each table lists the results for all pre-training datasets, along with their number of instances, their number of features, and their domain. The results are sorted in descending order of performance, highlighting which pre-training datasets achieve better performance on each benchmark. Table 3: AUC scores on CC-18 for TabDPT pre-trained on various datasets, sorted in descending order of AUC. Dataset APSFailure colleges ipums_la_99-small ipums_la_97-small ipums_la_98-small dionis cardiotocography Census-Income volkert KDDCup09_appetency covertype gas-drift-different-concentrations KDDCup09_churn road-safety kick eye_movements gas-drift philippine dilbert christine MiniBooNE Satellite jasmine scene helena kdd_internet_usage SpeedDating pol elevators musk one-hundred-plants-texture heloc default-of-credit-card-clients ada_agnostic sylva_agnostic colleges_usnews spoken-arabic-digit ada jannis PizzaCutter3 fbis.wc porto-seguro sylvine cjs Click_prediction_small guillermo

AUC on CC-18 # Instances # Features Dataset Domain 0.918 0.918 0.917 0.916 0.915 0.915 0.914 0.914 0.912 0.912 0.912 0.911 0.911 0.910 0.910 0.908 0.906 0.905 0.904 0.901 0.900 0.900 0.899 0.899 0.896 0.896 0.895 0.895 0.894 0.894 0.894 0.894 0.893 0.892 0.889 0.889 0.887 0.887 0.885 0.885 0.884 0.883 0.883 0.880 0.879 0.879

14

76000 7063 8844 7019 7485 416188 2126 299285 58310 50000 581012 13910 50000 111762 72983 10936 13910 5832 10000 5418 130064 5100 2984 2407 65196 10108 8378 10082 16599 6598 1599 10000 13272 4562 14395 1302 263256 4147 83733 1043 2463 595212 5124 2796 39948 20000

171 industrial-operational 45 other 57 financial-demographic 61 financial-demographic 56 financial-demographic 61 other 36 medical-human-sensor 42 financial-demographic 181 other 231 human-behaviour 55 biology-ecology 130 other-science 231 industrial-operational 33 human-behaviour 33 industrial-operational 28 medical-human-sensor 129 other-science 309 other 2001 other 1637 other 51 physics-astronomy 37 physics-astronomy 145 other 300 vision-audio-text 28 other 69 financial-demographic 121 human-behaviour 27 industrial-operational 19 other 168 other-science 65 biology-ecology 23 financial-demographic 21 financial-demographic 49 financial-demographic 217 biology-ecology 34 other 15 vision-audio-text 49 other 55 other 38 other 2001 vision-audio-text 58 human-behaviour 21 other 35 biology-ecology 12 human-behaviour 4297 other

Table 3: AUC scores on CC-18 for TabDPT pre-trained on various datasets, sorted in descending order of AUC. Dataset mushroom PieChart3 higgs magic ada_prior MagicTelescope eeg-eye-state JapaneseVowels shuttle ldpa okcupid-stem house_16H credit house_8L artificial-characters page-blocks sf-police-incidents albert walking-activity pbcseq fabert compas-two-years Diabetes130US madeline airlines analcatdata_halloffame visualizing_soil analcatdata_supreme chess kropt kr-vs-k poker-hand pollen puma8NH yeast hill-valley imdb.drama twonorm nursery gametes_epistasis mofn-3-7-10 parity5_plus_5

AUC on CC-18 # Instances # Features Dataset Domain 0.879 0.877 0.877 0.874 0.874 0.873 0.867 0.866 0.865 0.863 0.863 0.860 0.854 0.851 0.850 0.849 0.847 0.846 0.844 0.843 0.834 0.833 0.828 0.821 0.820 0.818 0.817 0.814 0.796 0.794 0.780 0.777 0.775 0.736 0.727 0.720 0.684 0.677 0.560 0.521 0.504 0.444

8124 1077 98050 19020 4562 19020 14980 9961 58000 164860 50789 22784 16714 22784 10218 5473 2215023 425240 149332 1945 8237 4966 71090 3140 539383 1340 8641 4052 28056 28056 28056 1025009 3848 8192 1269 1212 120919 7400 12958 1600 1324 1124

23 biology-ecology 38 other 29 physics-astronomy 11 physics-astronomy 15 financial-demographic 12 physics-astronomy 15 medical-human-sensor 15 vision-audio-text 10 physics-astronomy 8 medical-human-sensor 20 human-behaviour 17 financial-demographic 11 financial-demographic 9 financial-demographic 8 deterministic-simulated 11 vision-audio-text 9 human-behaviour 79 other 5 medical-human-sensor 19 medical-human-sensor 801 other 12 human-behaviour 8 medical-human-sensor 260 other 8 industrial-operational 17 other 5 biology-ecology 8 other 7 deterministic-simulated 7 deterministic-simulated 7 deterministic-simulated 11 deterministic-simulated 6 biology-ecology 9 deterministic-simulated 9 biology-ecology 101 deterministic-simulated 1002 vision-audio-text 21 deterministic-simulated 9 human-behaviour 21 deterministic-simulated 11 deterministic-simulated 11 deterministic-simulated

Table 4: R2 scores on CTR-23 for TabDPT pre-trained on various datasets, sorted in descending order of R2 . Dataset colleges kick ipums_la_97-small dionis ipums_la_98-small

R2 on CTR-23 # Instances # Features Dataset Domain 0.693 0.693 0.691 0.685 0.685

15

7063 72983 7019 416188 7485

45 other 33 industrial-operational 61 financial-demographic 61 other 56 financial-demographic

Table 4: R2 scores on CTR-23 for TabDPT pre-trained on various datasets, sorted in descending order of R2 . Dataset Census-Income APSFailure ipums_la_99-small KDDCup09_churn covertype road-safety gas-drift-different-concentrations gas-drift eye_movements musk KDDCup09_appetency volkert MiniBooNE sylvine cardiotocography dilbert default-of-credit-card-clients sylva_agnostic helena jannis pol scene philippine heloc one-hundred-plants-texture magic elevators MagicTelescope house_16H PieChart3 ldpa JapaneseVowels page-blocks christine spoken-arabic-digit higgs eeg-eye-state colleges_usnews jasmine SpeedDating Satellite cjs porto-seguro PizzaCutter3 Click_prediction_small ada_prior shuttle sf-police-incidents artificial-characters pbcseq airlines house_8L ada kdd_internet_usage

R2 on CTR-23 # Instances # Features Dataset Domain 0.684 0.682 0.681 0.679 0.679 0.678 0.677 0.672 0.672 0.671 0.671 0.669 0.668 0.667 0.665 0.665 0.664 0.657 0.655 0.649 0.647 0.647 0.646 0.645 0.643 0.643 0.641 0.640 0.639 0.638 0.635 0.634 0.633 0.633 0.631 0.630 0.630 0.629 0.627 0.627 0.623 0.621 0.620 0.617 0.615 0.612 0.608 0.607 0.604 0.585 0.585 0.581 0.581 0.579

16

299285 76000 8844 50000 581012 111762 13910 13910 10936 6598 50000 58310 130064 5124 2126 10000 13272 14395 65196 83733 10082 2407 5832 10000 1599 19020 16599 19020 22784 1077 164860 9961 5473 5418 263256 98050 14980 1302 2984 8378 5100 2796 595212 1043 39948 4562 58000 2215023 10218 1945 539383 22784 4147 10108

42 financial-demographic 171 industrial-operational 57 financial-demographic 231 industrial-operational 55 biology-ecology 33 human-behaviour 130 other-science 129 other-science 28 medical-human-sensor 168 other-science 231 human-behaviour 181 other 51 physics-astronomy 21 other 36 medical-human-sensor 2001 other 21 financial-demographic 217 biology-ecology 28 other 55 other 27 industrial-operational 300 vision-audio-text 309 other 23 financial-demographic 65 biology-ecology 11 physics-astronomy 19 other 12 physics-astronomy 17 financial-demographic 38 other 8 medical-human-sensor 15 vision-audio-text 11 vision-audio-text 1637 other 15 vision-audio-text 29 physics-astronomy 15 medical-human-sensor 34 other 145 other 121 human-behaviour 37 physics-astronomy 35 biology-ecology 58 human-behaviour 38 other 12 human-behaviour 15 financial-demographic 10 physics-astronomy 9 human-behaviour 8 deterministic-simulated 19 medical-human-sensor 8 industrial-operational 9 financial-demographic 49 other 69 financial-demographic

Table 4: R2 scores on CTR-23 for TabDPT pre-trained on various datasets, sorted in descending order of R2 . Dataset kropt analcatdata_halloffame albert chess ada_agnostic okcupid-stem guillermo kr-vs-k walking-activity credit compas-two-years visualizing_soil fabert analcatdata_supreme poker-hand Diabetes130US fbis.wc madeline pollen puma8NH twonorm yeast hill-valley mofn-3-7-10 mushroom parity5_plus_5 nursery gametes_epistasis imdb.drama

B

R2 on CTR-23 # Instances # Features Dataset Domain 0.578 0.575 0.575 0.574 0.571 0.567 0.565 0.560 0.558 0.558 0.557 0.545 0.544 0.543 0.512 0.500 0.483 0.478 0.424 0.367 0.316 0.278 0.221 0.131 0.112 -0.048 -0.228 -0.806 -16.712

28056 1340 425240 28056 4562 50789 20000 28056 149332 16714 4966 8641 8237 4052 1025009 71090 2463 3140 3848 8192 7400 1269 1212 1324 8124 1124 12958 1600 120919

7 deterministic-simulated 17 other 79 other 7 deterministic-simulated 49 financial-demographic 20 human-behaviour 4297 other 7 deterministic-simulated 5 medical-human-sensor 11 financial-demographic 12 human-behaviour 5 biology-ecology 801 other 8 other 11 deterministic-simulated 8 medical-human-sensor 2001 vision-audio-text 260 other 6 biology-ecology 9 deterministic-simulated 21 deterministic-simulated 9 biology-ecology 101 deterministic-simulated 11 deterministic-simulated 23 biology-ecology 11 deterministic-simulated 9 human-behaviour 21 deterministic-simulated 1002 vision-audio-text

Training and architecture details

Throughout this paper, “TabDPT” refers specifically to TabDPT v1.1. B.1

Base model: many-dataset pre-training ablations

This section provides the experimental setup for the many-dataset pre-training ablations. The deduplication, dataset-level filtering, and column-level pre-processing experiments all use the same base TabDPT architecture and training configuration described henceforth. Pre-training corpus We use part of the datasets collected by Hosseinzadeh et al. [18] as our pre-training corpus. This contains 1,732 datasets collected from OpenML [4], with datasets contained in the CC-18, CTR-23, or TabArena [8] benchmarks automatically removed. We do not filter datasets with fewer than 10 columns, unlike the cited paper, but we do exclude datasets with no clear target variable y. We additionally find that 197 of these datasets are exact duplicates. We cut off each dataset at 10,000 rows and 3,000 columns to bound pre-processing costs. Architecture We use a 12-layer transformer with embedding dimension dmodel = 512, 8 attention heads, and MLP hidden dimension 1,024 (hidden factor 2). The model uses post-layer normalization and an asymmetric in-context attention pattern: query (test) tokens attend only to context (train) tokens, never to each other. Feature columns are projected to dmodel via a shared linear encoder. For context tokens, the target embedding (a separate learned linear layer) is added on top of the feature embedding. At each step, inputs are capped at 100 feature columns and 10 output classes. 17

Training At each step, one column is selected uniformly at random as the prediction target y while the remaining columns form the feature matrix X. Classification and regression heads are trained jointly in a single forward pass using a self-supervised SSL objective. Label smoothing of 0.1 is applied to the classification cross-entropy loss. We use AdamW with learning rate 5 × 10−4 , weight decay 0.05, and no learning-rate schedule. Gradient norms are clipped to 1.0. Dropout is 0.0 throughout. Each run trains for 128,000 gradient update steps, with per-step batch size of 32 and gradient accumulation over 8 steps for an effective batch size of 256. The context window is randomly sampled at each step and is at minimum 50 tokens and at maximum 1,024. Data pre-processing and retrieval Each dataset is truncated to at most 10,000 rows and 3,000 columns before any pre-processing. Categorical columns are coerced to integers via label encoding and columns where fewer than 1% of values are interpretable as numeric are treated as categorical. Missing values are imputed with column means. The joint [X | y] matrix is then standardized with a StandardScaler. A FAISS flat L2 index is built on training-split embeddings and used for retrieval-augmented context assembly during training. Evaluation We report results on the same two standard benchmarks used previously: CC-18 for classification and CTR-23 for regression. On all metrics, we report the Interquartile Mean (IQM), which trims the bottom and top quartiles and is robust to outliers. We also report 95% bootstrap confidence intervals computed via stratified bootstrap over datasets using 20,000 resampling iterations. Inference uses context size 2,048 and 8 ensemble passes on the raw benchmark datasets, without applying any column pre-processing, to ensure consistency across experiments. Hardware Each run uses a single Nvidia A100 (40 GB) GPU with 8 CPU workers and 150GB RAM. B.2

Large model: context-length and architecture scaling

The experiments in subsection 3.3 use a larger model trained at three context lengths, with and without pre-processing, for six runs total. All six share the configuration described here. Architecture We increase the model to 16 transformer layers and embedding dimension dmodel = 768, with 8 attention heads and MLP hidden dimension 1,536 (hidden factor 2). Pre-layer normalization replaces the post-norm used in the base model. All other architectural choices (asymmetric attention, feature/label encoders, output heads, input caps) are unchanged. Training We use the ScheduleFree AdamW optimizer with learning rate 1 × 10−3 and weight decay 0.05. Training uses BF16 mixed precision. Gradient clipping (1.0) and label smoothing (0.1) are unchanged. Each run trains for 128,000 gradient steps. Batch size is 32 with gradient accumulation over 4 steps, giving effective batch size 128. The three variants differ in the maximum number of labelled context tokens (shown in the main text) but the minimum context size is 100 for all three. Inference for all six runs uses context size 2,048 with no column pre-processing applied, for consistency across comparisons. Data pre-processing and retrieval Dataset caps, categorical encoding, imputation, scaling, and FAISS L2 retrieval are identical to the base model. For the “+pp” variants, the NaN-column drop, Spearman correlation deduplication (threshold 0.90), and minimum-feature filter (k = 5) are applied to each dataset before training, with the column mask fitted on the training split as in the base ablations. Hardware Each of the six runs uses a single Nvidia H100 GPU. All other software and infrastructure settings are as described in Section B.1.

C

Additional TabArena results

We provide extended results on the TabArena benchmark [8]. All experiments use TabArena-Lite (51 tasks, fold 0), bootstrapped 2000 times. Elo is computed against a pool of default-configuration baselines (no tuning or ensembling), and is anchored at Random Forest = 1000. We evaluate six 18

Model A: Winner

Model B: Loser

O O Our Perp R ur_T ur_T O O _Tab Our Ran et _ M Tab eTaalTababDPT CabDPuTr_Taubr_Tab D PFN bIC PFN _4k atB _2k DPT DPT M Tab T Ta ReaPl T_1TkabPTaFbDPoTdern XGB LightTGorcdhomFo xFastai TabESxtrauTalBoo Lin -2.6 Lv2 -2.5 _pp oost _pp _4k _2k itra DPT abM bICL MLP _pp Nv2 _1k NCA oost EBM BM MLP rest RFM MLP TAR rees ster ear KNN (de (de (de (def (de (def (def (def (de (de (de (de (de (def (de (def (de (de (de (de (de (de (de (de (de (de (de (de (de faul faul faul aul faul aul aul aul faul faul faul faul faul aul faul aul faul faul faul faul faul faul faul faul faul faul faul faul faul t) t) t) t) t) t) t) t) t) t) t) t) t) t) t) t) t) t) t) t) t) t) t) t) t) t) t) t) t) 49 69 82 88 80 80 82 90 78 90 90 92 86 82 92 96 96 90 96 98 98 96 98 88 98 98 98 100 TabPFN-2.6 (default) 61 76 82 78 80 78 82 80 90 84 86 88 84 88 92 86 92 92 96 94 92 98 94 94 96 100100 TabICLv2 (default) 51 65 73 69 73 73 80 76 80 75 82 78 92 78 88 88 88 96 98 94 92 96 92 96 94 98 98 RealTabPFN-2.5 (default) 31 39 51 57 67 67 55 59 53 49 57 78 61 82 71 63 63 69 80 84 88 88 84 92 92 88 100 Our_TabDPT_4k_pp (default) 18 24 35 55 57 55 45 61 55 57 69 61 61 59 67 78 75 80 92 92 84 94 86 88 94 98 100 CatBoost (default) 12 18 27 49 59 61 51 55 53 51 59 75 59 80 67 61 63 69 78 80 84 86 82 92 90 88 100 Our_TabDPT_2k_pp (default) 20 22 31 43 45 53 49 53 51 49 55 65 53 75 69 65 63 69 76 78 86 82 82 92 86 88 100 Our_TabDPT_4k (default) 20 20 27 33 43 41 47 49 57 49 53 65 57 80 67 63 59 67 78 76 84 82 78 92 88 88 100 Our_TabDPT_2k (default) 18 22 27 33 45 39 47 51 59 55 59 51 59 57 61 61 67 63 71 78 80 82 78 86 88 90 100 Mitra (default) 10 18 20 45 55 49 51 53 53 43 49 59 55 69 65 59 55 69 78 78 80 86 76 92 86 86 98 TabDPT (default) 22 20 24 41 39 45 47 51 49 53 59 53 57 55 61 69 57 73 86 82 76 86 88 82 88 92 96 TabM (default) 10 10 20 47 45 47 49 43 41 47 53 61 57 61 58 61 63 65 75 81 67 82 82 75 86 98 100 TabICL (default) 10 16 25 51 43 49 51 51 45 57 47 53 57 49 59 67 59 61 86 80 76 94 88 82 92 98 100 RealMLP (default) 8 14 18 43 31 41 45 47 41 51 41 47 51 63 59 59 59 65 78 80 82 80 78 88 88 88 100 Our_TabDPT_1k_pp (default) 14 12 22 22 39 25 35 35 49 41 47 39 47 47 43 47 55 51 65 71 75 76 73 78 82 90 98 TabPFNv2 (default) 18 16 8 39 39 41 47 43 41 45 43 43 43 49 61 57 57 59 75 78 84 78 78 86 86 86 100 Our_TabDPT_1k (default) 8 12 22 18 41 20 25 20 43 31 45 39 51 37 53 47 45 59 84 86 73 84 82 82 88 90 98 ModernNCA (default) 4 8 12 29 33 33 31 33 39 35 39 42 41 41 57 39 57 53 75 78 75 76 78 76 86 88 96 XGBoost (default) 4 14 12 37 22 39 35 37 39 41 31 39 33 41 53 43 53 49 78 67 67 78 84 73 82 96 96 EBM (default) 10 8 12 37 25 37 37 41 33 45 43 37 41 41 45 43 55 43 71 76 73 78 80 80 80 90 94 LightGBM (default) 4 8 4 31 20 31 31 33 37 31 27 35 39 35 49 41 41 47 51 59 55 55 71 67 63 84 92 TorchMLP (default) 2 4 2 20 8 22 24 22 29 22 14 25 14 22 35 25 16 25 22 29 47 55 59 59 59 75 98 RandomForest (default) 2 6 6 16 8 20 22 24 22 22 18 19 20 20 29 22 14 22 33 24 41 49 61 55 55 65 88 xRFM (default) 4 8 8 12 16 16 14 16 20 20 24 33 24 18 25 16 27 25 33 27 45 53 67 53 67 75 90 FastaiMLP (default) 2 2 4 12 6 14 18 18 18 14 14 18 6 20 24 22 16 24 22 22 45 45 51 49 53 65 80 TabSTAR (default) 12 6 8 16 14 18 18 22 22 24 12 18 12 22 27 22 18 22 16 20 29 41 39 33 57 69 96 ExtraTrees (default) 2 6 4 8 12 8 8 8 14 8 18 25 18 12 22 14 18 24 27 20 33 41 45 47 51 65 84 PerpetualBooster (default) 2 4 6 8 6 10 14 12 12 14 12 14 8 12 18 14 12 14 18 20 37 41 45 33 47 43 67 Linear (default) 2 0 2 12 2 12 12 12 10 14 8 2 2 12 10 14 10 12 4 10 16 25 35 25 35 31 35 KNN (default) 0 0 2 0 0 0 0 0 0 2 4 0 0 0 2 0 2 4 4 6 8 2 12 10 20 4 16 33

Win Rate

100%

80%

60%

40%

20%

0%

Figure 10: Pairwise win-rate matrix on TabArena-Lite across all evaluated methods (our models and baselines). Each cell shows the fraction of tasks where the row method outperforms or ties the column method. TabDPT variants spanning a 2 × 3 grid over pre-processing (+pp vs. none) and training context length (1k, 2k, 4k tokens), alongside a broad set of baseline methods. C.1

Pairwise win rates

Figure 10 shows pairwise win rates across all methods. The relative ordering among our variants matches the main text: 4k+pp performs best, followed by 2k+pp and 4k, with shorter-context models trailing. C.2

Leaderboard results

Table 5 reports leaderboard metrics for all evaluated methods. The 4k+pp variant achieves the strongest performance among our models, and the remaining variants follow the same ordering as in Figure 10.

D

Attention similarity experiments: methods and supplementary results

D.1

Models, probes, and similarity calculations

Checkpoints We probe 52 TabDPT checkpoints, each pre-trained from scratch on a single OpenML dataset, all sharing the same architecture (12 transformer blocks, dmodel = 512, single attention head, asymmetric in-context attention pattern) and the same training schedule (lr 5 × 10−4 , weight decay 0.05, sequence length 1536, taken at 12,800 gradient updates with an effective batch size of 256). Probes Each model is evaluated on two held-out OpenML classification datasets, A MA ZON _ EMPLOYEE _ ACCESS (OpenML ID 4135) and NURSERY (OpenML ID 1568), that none of

19

Table 5: Full TabArena-Lite leaderboard (51 tasks, fold 0). Elo is anchored at Random Forest = 1000. Our models are in bold. Method Elo Avg. Rank TABPFN-V2.6 TABICLv2 REALTABPFN-V2.5 Our TabDPT (4k+pp) CatBoost Our TabDPT (2k+pp) Our TabDPT (4k) Our TabDPT (2k) MITRA_GPU TabDPT_GPU TabM_GPU TabICL_GPU RealMLP_GPU Our TabDPT (1k+pp) TabPFNv2_GPU Our TabDPT (1k) ModernNCA_GPU XGBoost EBM LightGBM NeuralNetTorch Random Forest (anchor) xRFM_GPU FastAI TabSTAR ExtraTrees PerpetualBooster Linear Model KNN

1583 1545 1477.6 1322.8 1320.8 1299.8 1274.3 1270.7 1265.8 1259.1 1258.4 1257.2 1245.1 1220.0 1195.6 1190.8 1187.0 1179.5 1177.1 1151.5 1024.4 1000.0 999.6 973.1 954.4 945.9 908.8 816.1 583.2

4.16 4.80 6.16 10.25 10.31 10.96 11.76 11.88 12.04 12.25 12.27 12.31 12.71 13.53 14.33 14.49 14.62 14.86 14.94 15.78 19.75 20.44 20.45 21.18 21.67 21.88 22.78 24.71 27.71

the 52 models was trained on. For each (model, probe) we record the per-block attention tensor of shape (B, H, Q, C) = (32, 1, 512, √512) at a fixed in-context cutoff eval_pos = 512, where αb,h,q,c = softmaxc (⟨qb,h,q , kb,h,c ⟩/ dhead ) is a probability vector over the C context tokens (the attention distribution of query q). Pair similarity

For two models i, j on probe p and block ℓ we compute X (i) (j)
(ℓ,p) 1 sij = BQ ρSpearman αb,q,· , αb,q,· , b,q

which is the row-wise Spearman rank correlation between the two attention distributions, that is then (ℓ,p) averaged over queries and probe examples. The pair-level scalar in the main paper averages sij over the 12 blocks. Spearman is preferred over Pearson because rank correlation is invariant to the exact magnitudes of the attention values. Held-out AUC Per-model AUC is the AUC on the OpenML CC-18 classification suite at the same training step, which is data disjoint from both the training datasets and the two probes. D.2

Strong / weak partitioning

We split the 52 models by AUC quartile (q25 = 0.831, q75 = 0.900). Models with AUC ≥ q75 are strong (S, 13 models), models with AUC ≤ q25 are weak (W, 13 models). The 26 middle-quartile models are excluded.

20

Table 6: Models in the strong and weak sets, identified by their training dataset. Strong (AUC ≥ 0.900) Weak (AUC ≤ 0.831) APSFAILURE, C ENSUS -I NCOME, KDDC UP 09_ APPETENCY, S PEED DATING, CARDIOTOCOGRAPHY , C OLLEGES, COVERTYPE, GAS - DRIFT, IPUMS _ LA _98- SMALL, MUSK, POL, ROAD - SAFETY , SCENE

D IABETES 130US, AIRLINES, ANALCATDATA _ SUPREME , CHESS, HILL - VALLEY , PARITY 5_ PLUS _5, PBCSEQ , POKER - HAND , POLLEN, PUMA 8NH, TWONORM , VISUALIZING _ SOIL, YEAST

All 95% CIs in Tables 2 and 7 are obtained by bootstrap. We repeatedly draw a version of each group by sampling its observations with replacement, recompute the gap between the two group means, and collect the resulting gap estimates. The 95% CI is the range from the 2.5th to the 97.5th percentile of that collection. We use 10 000 resamples for Table 2 and 2 000 for Table 7. The random seed 0 is fixed for reproducibility. D.3

Per-block breakdown

Table 7 reports, per transformer block, the SS / SW / WW group means together with the SS−SW gap and its 95% bootstrap CI (K = 2 000). Each row pools across the two probes, so nSS = 156, nSW = 338, nWW = 156 per block. Table 7: Per-block group means and SS−SW gap with 95% bootstrap CI. One observation per (model pair, probe, layer). Rows whose CI excludes zero are marked †. Block SS SW WW SS−SW 95% CI 0 1 2 3 4 5 6 7 8 9 10 11

0.854 0.525 0.123 0.145 0.551 0.584 0.577 0.303 0.520 0.585 0.617 0.667

0.696 0.442 0.047 0.110 0.383 0.363 0.403 0.295 0.368 0.363 0.213 0.407

0.593 0.402 0.279 0.156 0.330 0.238 0.277 0.325 0.269 0.253 0.073 0.288

+0.158† +0.083† +0.075 +0.035 +0.168† +0.221† +0.174† +0.008 +0.153† +0.222† +0.404† +0.260†

[+0.124, +0.193] [+0.005, +0.164] [−0.027, +0.176] [−0.050, +0.119] [+0.132, +0.205] [+0.187, +0.255] [+0.133, +0.218] [−0.057, +0.075] [+0.124, +0.181] [+0.197, +0.249] [+0.367, +0.439] [+0.239, +0.284]

The SS−SW gap is positive in all 12 blocks, statistically significant in 9 of them (blocks 0, 1, 4, 5, 6, 8, 9, 10, 11), and largest at block 10. We also see the global correlation effect shown in Figure 9, being concentrated in the deeper transformer blocks in Figure 11: per-block correlation rises from near zero in early blocks to ≈ 0.6 in blocks 9–11. D.4

Caveat: pair non-independence

The data points are not fully independent, since each model appears in 51 pairs, so the confidence intervals reported above are somewhat tighter than they would be under a stricter resampling scheme.

E

Additional query–key sharing ablation

Figure 12 is an additional analysis of training dynamics when WQ = WK is enforced, on two further pre-training tables that generalize poorly, the V ISUALIZING S OIL and the C HESS datasets. In both cases, forcing WQ = WK yields significantly higher validation AUC than the default architecture, 21

Figure 11: Per-block pair attention similarity vs. mean AUC, n = 2652 points (one per model pair × probe dataset).

Figure 12: Ablations where WQ = WK is enforced, on V ISUALIZING S OIL (left) and C HESS (right), using the same setting as Figure 7a. complementing the main-text weak-table results of Figure 7a and supporting the view that TFM performance is close to a nearest neighbour-like mechanism.

22

Related documents

• Do Tabular Foundation Models Agree with Themselves?#432550
• Statistical and Optimization Principles for Understanding and Controlling Foundation Models#634883
• From Synthetic Priors to Model Behavior: Structural Coverage in Tabular Foundation Models#974991
• SkillTFM: Gated Skill Evolution for Training-Free Adaptation of Tabular Foundation Models#432538
• Do Tabular Foundation Models Still Need Feature Engineering?#1071009