Lennart Purucker1,2 Andrej Tschalzev3 Nick Erickson1 Gioia Blayer4 David Holzmüller4 Alan Arazi5,1 Alexander Pfefferle6,2 Mustafa Tajjar2,7 Gaël Varoquaux4,8 Frank Hutter1,6,2 1 2 Prior Labs University of Freiburg 3 University of Mannheim 4 INRIA Saclay 5 Technion 6 ELLIS Institute Tübingen 7 Zuse School ELIZA 8 Probabl [email protected]
Abstract Foundation models for predictive machine learning on tabular data have recently gained significant traction in academia and industry. Research communities across disciplines are increasingly evaluating tabular foundation models on diverse datasets and tasks. However, these task- and discipline-specific evaluations remain largely inaccessible to model researchers because benchmark software and evaluation protocols are fragmented. As a result, model researchers rely on standard benchmarks, which are mostly defined for tasks where tabular foundation models already excel. The most challenging scenarios are excluded, limiting meaningful progress in the field by focusing on marginal improvements on IID data rather than on broader, more demanding challenges. To overcome this, we introduce BeyondArena, the first unified holistic benchmark for tabular data that supports diverse task types (IID, temporal, grouped), across sample size and feature dimensionality scales, with diverse feature types (with text, with high cardinality) from a broad range of disciplines. To enable unified benchmarking beyond standard benchmarks, we introduce DataFoundry, a Python framework and metadata schema for curating tabular datasets for predictive machine learning. Our results across 11 models and 142 curated datasets show that existing tabular foundation models excel on tiny- to medium-sized IID data, while traditional tree-based and deep learning models still dominate on non-IID, large, and high-dimensional datasets. BeyondArena guides model research for the most demanding challenges in tabular data, enabling progress towards truly foundational tabular models. Pipeline: Family:
Elo (Best Model)
arXiv:2606.30410v1 [cs.LG] 29 Jun 2026
Beyond IID: How General Are Tabular Foundation Models, Really?
1500 1400 Task 1300 1200 1100 1000 900 800 700 600 500
IID
N=103
ICL Default Tuned + Ensembled MLP (RealMLP, TabM) TFM (TabICLv2, TabPFN-2.6, TabDPT)
GBDT (CatBoost, XGBoost, LightGBM)
Scale
Grouped Temporal N=18
N=21
Tiny
N=52
Baseline (RandomForest, ExtraTrees, Linear)
Dimensionality
Small N=38
Medium N=30
Large N=22
Low
N=91
High N=51
Features
Text
N=16
High Card. N=30
All
N=142
Figure 1: BeyondArena Results. We show the Elo of the best model per family across subbenchmarks, comparing state-of-the-art MLPs, gradient-boosted decision trees (GBDT), tabular foundation models (TFM), and simple baselines. For TFMs, we evaluate in-context learning (ICL) performance. For traditional models, we evaluate performance with default hyperparameters and with tuning plus post-hoc ensembling. We show results per model in Figure 5.
Preprint.
1
Introduction
Tabular foundation models (TFMs) have been evaluated across many research communities, ranging from graph learning [1–4] or surrogate modeling [5–7], to real-world applications such as Cybersecurity [8], soil mapping [9], or clinical predictions [10]. While the broader research community evaluates TFMs across diverse tasks and datasets, the tabular research community is so far limited to evaluations on IID tasks [11–13]. Moreover, historically, projects that broaden evaluation diversity were often developed with independent benchmarking infrastructure and scientific standards, such as PMLB-Mini [14] for tiny data, TabReD [15] for tabular-temporal tasks, or TextTabBench [16] for tabular-text data. The current state of benchmarking for machine learning on tabular data is not only confusing to practitioners but also impeding researchers. We aim to unify benchmarking and accelerate research on TFMs. Thus, we introduce BeyondArena, a holistic benchmark for machine learning on tabular data that unifies independent and identically distributed (IID) and non-IID task types (IID, temporal, grouped) across datasets with varying sample scales (100-1k, 1k-10k, 10k-100k, 100k to 1 million), dimensionality scales (≤100, >100), with challenging feature types (high-cardinality categories, or text) from a broad range of applied disciplines. To create BeyondArena, we curate a large collection of representative datasets and provide reliable baselines. Furthermore, we unify the tabular benchmarking community by integrating our benchmark into TabArena’s [11] open-source ecosystem. To summarize, our contributions are: ❖ We manually curate 142 datasets from 1128 – following rigorous protocols – to obtain highquality tabular datasets across scales and feature types, and publish DataFoundry, a Python framework and metadata schema for reproducible tabular data curation; ❖ We systematically evaluate whether 3 state-of-the-art open-source TFMs generalize beyond IID settings by comparing them with 8 strong traditional baselines across diverse tasks; ❖ We integrate all our contributions into TabArena’s open-source benchmark ecosystem, thereby avoiding fragmentation of the research community: code at https://tabarena.ai/code, data at https://github.com/TabArena/data-foundry. Our driving motivation behind BeyondArena is to understand how well TFMs would perform when practitioners use them in real-world predictive applications, thereby closing the gap between evaluations in academia and real applications. In this paper, we detail our dataset curation, standardize benchmarking across task types, and evaluate the strengths and weaknesses of existing TFMs. BeyondArena focuses on evaluating predictive machine learning models for tabular classification and regression on non-IID data, ranging from tiny to large (100-1M), with diverse feature types (high-cardinality categorical and text features). Out of scope are few-shot predictions (<100), images as features, and other tasks such as survival analysis or relational learning. Our results demonstrate that state-of-the-art TFMs dominate on tiny, small, and IID data, while they fail to compete with traditional tree-based and deep learning models on non-IID (temporal, grouped), large-scale, high-dimensional, and high-cardinality categorical datasets. We further ablate our experimental setup to assess the validity of our claims and show the importance of (i) appropriate outer test splits for grouped data; (ii) appropriate inner validation splits for non-IID or tiny data; (iii) preprocessing for grouped data and text features; (iv) and calibrating probabilities.
2
Background: IID and non-IID Tabular Data
Tabular foundation models and, in extension, tabular deep learning have been evaluated in the broader research community across countless applications, from computer science disciplines (cf. [1–7, 17– 36]) to applied science disciplines (cf. [8–10, 37–74]). To categorize such broad application data, we adopt an application-dependent definition of independent and identically distributed (IID) and non-IID data. We determine whether a dataset is IID or non-IID based on the appropriate train-test split. We deem the split appropriate that most closely mirrors the original real-world application. Illustration. Two practitioners could work with the same data but for fundamentally different applications. Practitioner Alice uses the data to predict whether past transactions were fraudulent for follow-up fraud investigations. Practitioner Bob uses the data to predict whether new transactions are fraudulent, aiming to prevent fraud in real time. To estimate model performance, Alice uses a random IID split, while Bob uses a non-IID temporal split to simulate the expected distribution of unseen transactions when the model is deployed. If Bob were to use a random split, it would lead 2
to overestimating the performance of models that can exploit temporal leakage [15, 75]. On the contrary, Alice cannot use a temporal split, as this would underestimate the performance of such models. The appropriate split depends on the practitioner’s application. BeyondArena extends this line of thinking from practitioners to data curation and academic benchmarking. IID Tabular Data. We define data as IID if the test samples in the associated application do not follow a particular structure, allowing a random split. That is, we hold out randomly sampled data. Non-IID Tabular Data. We define data as non-IID when the application requires a temporal or grouped split. In a temporal split, a time index is required, and test samples occur strictly after the training data, reflecting prediction for observations in the future (e.g., future transactions). In a grouped split, a group index is required, and all samples with the same index are kept together, so no group appears in both the training and test sets. Such applications aim to generalize to samples associated with unseen entities (groups). Grouped tasks can be of type label-per-group, where all samples in a group share the same label, or label-per-sample, where each sample has its own label. For label-per-group, the objective is to predict a group-level label (e.g., if a customer is fraudulent based on a collection of transactions). For label-per-sample, the objective is to predict individual sample labels for data sources not seen during training (e.g., a new country or hospital). For some datasets, either a temporal or a grouped split is plausible. Using a temporal split does not remove group structures, and using a grouped split does not make the task time-invariant. The split only determines the importance of grouped or temporal dependencies – it does not remove them. BeyondArena does not include data from time-series forecasting applications, despite the data being tabular-like and non-IID [76, 77]. During data curation, we distinguish data stemming from temporal tabular regression and time-series forecasting tasks, as they have fundamentally different assumptions, necessitating different validation procedures; see Appendix A.1. Nevertheless, tabular models can be used for time-series forecasting tasks [21, 78–80], and can be competitive [81–84].
3
Related Work
As a unified tabular benchmark, BeyondArena relates to IID and non-IID tabular benchmarks. IID Tabular Benchmarks. The vast majority of prior benchmarks for predictive machine learning on tabular data focused on IID tasks, often inappropriately treating all data as stemming from IID applications. Benchmarks were created during model development [85–88], for tabular deep learning [89–96], to develop AutoML systems [97–102], or from a data-centric view [103–110], all aiming to understand and compare models. Recent benchmark projects have focused on broadening evaluation diversity by supporting few-shot or tiny data [14, 111], including diverse domains and large data [12, 112], or by curating datasets representative of real-world tasks [11]. BeyondArena builds on prior benchmarking efforts by extending their high-quality work beyond IID data. In parallel, the research community developed benchmarks for multimodal tabular data. These efforts concentrated on tabular data with text [16, 113–121] or to benchmark LLMs solving predictive tasks by treating tables as text [122–135]. Tabular data with images has received little benchmarking attention [116, 136, 137], yet there have been application-specific model comparisons [115, 117, 138–147]. Time-series classification or regression benchmarks [148–151] are also multimodal IID benchmarks. They use random splits, and the state-of-the-art includes tabular models with feature engineering [152–154] or pretrained encoders [155, 156]. BeyondArena makes a first step toward unifying multimodal benchmarking by incorporating IID and non-IID tabular data with text. We leave tabular data with images and data from time-series classification or regression for future work. Non-IID Tabular Benchmarks. Benchmarks for non-IID tabular data focused mainly on temporal data. While the broader research community worked on non-IID tabular data for decades [157–164], only recently Rubachev et al. [15] identified temporal tabular data as a gap in tabular deep learning benchmarks and introduced TabReD, enabling new research [165, 166]. BeyondArena builds upon TabReD by supporting more temporal datasets, more baselines, and tabular foundation models. Moreover, BeyondArena enables research on IID and temporal tabular data in a single framework. There have been no dedicated benchmarks for tabular deep learning on grouped data. Many applications of grouped tabular data originate from longitudinal data [56, 167–170], algorithm selection problems [171–173], or distribution shift benchmarks [174–179]. BeyondArena is the first benchmark to incorporate grouped data from all of these fields. Thus, BeyondArena fills a gap in non-IID benchmarking and enables new research on models that generalize across diverse application domains. 3
To summarize, compared to related work, we unify IID and non-IID benchmarking in BeyondArena by curating representative datasets in one shared metadata format (Section 4) and enabling rigorous evaluation of state-of-the-art tabular models on all tasks and multimodal data types (Section 5).
4
Dataset Curation
Our curation process aims to extend tabular benchmarking to better represent more challenging realworld predictive machine learning applications. Therefore, we overhaul the selection criteria from TabArena [11] and extend them to BeyondArena’s focus by supporting: (I) non-IID data (temporal and grouped); (II) data with less than 500 or more than 100 000 training samples; and (III) tabular data with text and date features. To ensure selected datasets are high-quality and representative of BeyondArena’s focus, we opted for a manual curation process in which all results were verified by humans1 . While this choice was extremely labor-intensive, it is crucial to the scientific rigor, as the data quality directly affects the correctness of the benchmark’s conclusions [15, 75, 109, 110]. Data Foundry. To enable such extensive dataset curation and make it reproducible, we introduce DataFoundry, a Python framework and metadata schema for tabular datasets and predictive machine learning: https://github.com/TabArena/data-foundry. DataFoundry contains: (1) a mature Python package for dataset curation, and (2) the exact code we used to process each dataset in easy-to-understand Python notebooks (.ipynb). The Python package delivers code for dataset checks, train-test splits, and an extensible metadata schema. In addition, DataFoundry has one Python notebook for each dataset in BeyondArena. Each notebook contains reproducible code for metadata edits, preprocessing, dataset checks, train-test splits, and exporting the artifact for consumption by benchmarks, data repositories (such as OpenML, Kaggle, or Hugging Face), or other applications. Dataset Sources. We gathered datasets from 21 tabular benchmark studies and searched for new datasets in public data repositories. In particular, we re-evaluated 304 datasets from 14 prior tabular benchmarks considered for the TabArena curation, comprising all accepted datasets and those rejected for being non-IID, too small, or for other data issues. We further collected datasets from 7 non-IID or multimodal tabular benchmarks: TabReD [15], TableShift [177], the string vectorizing benchmark [114], the CARTE benchmark [118], TextTabBench [16], the TabSTAR benchmark [180], and the AutoGluon Multimodal benchmark [116, 137, 181]. Furthermore, we searched for new datasets by browsing data repositories or competition websites (UCI Machine Learning Repository [182], OpenML [183, 184], Hugging Face [185], Kaggle [186], Zindi [187], ASlib [172]) or public domain government websites. Our search was guided by popularity, task, and application domain, while avoiding obvious duplicates, to provide diverse input for our curation process. Dataset Selection Criteria. A dataset had to fulfill the following criteria to become part of BeyondArena: (1) The dataset and its predictive machine learning task are unique within our benchmark; (2) The dataset is not a few-shot prediction task, that is, it has at least 100 train samples; (3) The dataset is from a tabular IID or non-IID task, that is, a random, temporal, or grouped split is the appropriate validation protocol; (4) The dataset was published explicitly for a predictive classification or regression task; (5) The dataset and its task are representative of a problem in a real-world application that a practitioner would solve with tabular machine learning models, that is, the dataset stems from a real random distribution (e.g., the dataset is not generated from a deterministic function), and tabular models are a suitable, competitive choice (e.g., the dataset is not a vectorized version of ImageNet). (6) The dataset and its predictive task do not raise (obvious) ethical concerns. Compared to the criteria by Erickson et al. [11], we made bigger but also nuanced changes. The bigger changes are that we allow non-IID data, relax the modality requirements, and allow the use of larger datasets with subsampling, making any dataset with at least 100 training samples in scope. Nuanced changes include a more precise definition of duplicates, a clearer distinction from tabular-adjacent predictive tasks, and a sharper definition of what constitutes a representative dataset for predictive classification or regression tasks. Furthermore, although some of our datasets overlap with the inclusion criteria of the TabArena datasets, we show that ours are more challenging (Appendix D.3). We provide a detailed description in Appendix B.2 and share our curation insights per dataset at https://tabarena.github.io/data-foundry/. Dataset Processing. Each selected dataset was thoroughly investigated and integrated into DataFoundry. By doing so, we unified the data format (e.g., .csv, .parquet, or .xlsx) and feature 1We investigated (semi-)automated agentic procedures with limited success; see Appendix B.1 for details.
4
(0) Dataset Discovery
(1) No duplicates
(2) No few-shot prediction task
(3) Must be from a predictive task
(4) Represents a real application -171 unsuited modality -115 artificial datasets -89 data quality issues -29 trivial tasks -6 ethical concerns
1128 datasets
-369 datasets
from 21 benchmarks and public data repositories
759 remain
-11 datasets
-196 datasets
748 remain
142 tiny to largesized, tabular IID and non-IID tasks
-410 datasets
552 remain
142 remain
Figure 2: Dataset Selection Results. We present the main reasons for filtering datasets (1-4). Within (4), we also show sub-categories. Roughly 12.6% of all investigated datasets were selected. 109
104
Task Type IID Grouped Temporal
10
106
102 105
101
Number of Datasets
107
103
Number of Cells
Number of Columns
108
103
105
104
6
4
2
104
102
8
0 1985
106
1990
1995
2000
2005
Number of Rows Numeric Text
Categorical Date/Time
2010
2015
2020
2025
Year Binary
100%
Share of columns
Problem
Binary Classification
Regression
73 (51%)
44 (31%)
Multiclass Classification 25 (18%)
75% Task
IID
Grouped
Temporal
103 (73%)
18 (13%)
21 (15%)
50% 63 (44%)
Domain
38 (27%)
0%
Datasets
48 (34%)
Business & Marketing
Medical & Healthcare
25%
0%
Kaggle
UCI
Source
33 (23%)
25%
Other (OpenML, ...)
Finance
Biology
18 (13%)
13 (9%)
50%
31 (22%)
Other (8 domains) 40 (28%)
75%
100%
Share of Datasets
Figure 3: Dataset Dashboard. (Top Left): Dataset sizes (w.r.t. rows, columns, and cells). (Top Right): Age distribution. (Bottom Left): Distribution of feature types per dataset. (Bottom Right): The share of datasets from a specific problem type, task type, dataset source, or application domain.
type annotations (numbers, categorical, dates, strings). Moreover, we store all necessary metadata for the task (e.g., train-test splits, target column, columns for the group or time index), as well as annotations from our curation process (e.g., notes on data anomalies), in a machine-readable data schema. Whenever necessary, we also apply feature engineering to create the appropriate task, following the task description in the paper or the top solutions on competition websites. We share details and some general guidelines that we developed during the curation process in Appendix B.3. Curation Outcome. We collected 1128 datasets, of which 142 met our selection criteria. As shown in Figure 2, the majority of datasets were duplicates, did not represent a real application, or were not originally from a predictive task. We summarize the characteristics of our dataset collection in Figure 3, with extended overviews in Appendix B.4. BeyondArena includes datasets that range from tiny to large-scale, are recent, and are diverse across feature types, problem types, task types, sources, and application domains. We provide details per dataset in Appendix B.5 and share their DataFoundry artifacts at https://huggingface.co/datasets/TabArena/BeyondArena.
5
Experimental Design
We follow best practices for evaluating tabular machine learning models. We compare state-of-theart models with verified open-source implementations, perform reliable validation and tuning, use industry-standard preprocessing, employ robust metrics, and rigorously repeat our experiments. BeyondArena Models. We compare 11 tabular machine learning models: Linear / Logistic Regression, Random Forest [188], Extremely Randomized Trees [189], CatBoost [190], LightGBM [191], XGBoost [192], RealMLP [85], TabM [87], TabDPT [193], TabPFN-2.6 [194, 195], and TabICLv2 [196]. We select models that performed competitively on the TabArena live leaderboard [197], TALENT [112], and TabReD [15], including a linear model, tree-based models, neural 5
networks, and TFMs. We run TabPFN-2.6 and TabDPT only for datasets with up to 100k training samples. For TabPFN-2.6, we stay within its designated pretraining limits. We limit TabDPT due to the infeasible cost of retrieval-based inference for benchmarking: for a dataset with 1 million samples under cross-validation, TabDPT needs 8 million forward passes, 8 per sample. We imputed missing results for TabPFN-2.6 and TabDPT using the performance of a default Random Forest. Open-source Implementation. We build BeyondArena on top of TabArena [11], using its model implementations and benchmark infrastructure. As a result, all models we use were unit-tested, implemented in the standardized AutoGluon-based model framework [181], and are open-source – ensuring high-quality, reproducible benchmarking. We modified implementations to scale them to larger data and ensure proper behavior for non-IID tasks (e.g., handling unseen categories). Moreover, we added non-IID validation protocols and new (multimodal) preprocessing to TabArena. Outer Splits. We create train-test splits for a dataset differently across sample sizes to guard against noise biasing the results while remaining conservative with the compute budget. We split and subsample datasets with more than 1.25 million samples into 1 million training and 250k test samples. For IID tasks, we use repeated 3-fold cross-validation for datasets with fewer samples. We perform n-repeated cross-validation with n = 3 for 2500-250k training samples; n = 10 for 500-2500; and n = 20 for fewer than 500. For grouped tasks, we do the same but utilize group-based cross-validation. We stratify splits on the target variable for IID and grouped classification tasks. For temporal tasks, we manually create application-specific temporal splits. The application determines the appropriate prediction time horizon for the test data. We roll back the time horizon to create multiple split time points for multiple train-test splits. At each time point, we use all data before for training and all data after within the time horizon for testing. We create the same number of splits as for IID or grouped tasks, but we do not use splits with < 50% of the original data as training data. We deem a split with < 50% too unrepresentative of the original application. Inner Validation and Tuning Protocol. After the outer train-test split, we split the training data into 8 folds for cross-validation (CV) to estimate model performance during tuning. For datasets with < 500 training samples, we use 5-repeated 5-fold CV to avoid overtuning [198, 199]. We stratify the splits for classification. For IID tasks, we use random splits. For grouped tasks, we use grouped splits based on the group index. For temporal tasks, we bin the time index and create intervals used as folds for CV, a popular procedure on Kaggle, and following Bergmeir and Benítez [159]. We tune models using the search spaces by Erickson et al. [11], due to a limited compute budget, with 25 configurations from random search. For TFMs (TabDPT, TabPFN-2.6, TabICLv2), we evaluate only in-context learning without (fine-)tuning to assess their pretrained generality using the default preprocessing pipeline and no dataset-specific weight updates. We set a 4-hour time limit for CV, plus 1 hour of overhead for predicting on test data. We increase the time limit to 12 hours for TabPFN-2.6 and TabICLv2, as the 5-hour limit was insufficient for larger datasets. We also had to reduce TabICLv2’s ensemble member count to 1 for the 5 largest datasets. Unlike traditional models, existing TFMs do not support early stopping to respect predefined time constraints [200]. Only ∼0.01% of jobs exceed 3.5 hours; see Appendix C.1 for details. Multimodal and non-IID Preprocessing. We adjust TabArena’s preprocessing pipeline to handle multimodal non-IID data. We use a single general preprocessing pipeline (model-agnostic) and extensions to it per model (model-specific). We retain the model-specific pipelines from TabArena, which handle, when needed, steps such as scaling or categorical encoding. Our new model-agnostic preprocessing pipeline handles date encoding, text encoding, and preprocessing for grouped data. We use skrub [201] to convert datetime features into 10 numerical features, representing the weekday or a spline-based periodic encoding. We encode text features using Qwen3-Embedding-8B [202], the best zero-shot multilingual text encoding model based on the MMTEB leaderboard [203, 204], into a 32-dimensional vector using Qwen3’s smallest Matryoshka representation learning [205] slice. Lastly, we add preprocessing for grouped non-IID data. For label-per-sample datasets, we follow common practice and drop the group-index column to prevent models from overfitting to it during training. For label-per-group datasets, we replace the group index with a 50-dimensional group-encoding vector, similar to common practice in Kaggle competitions (e.g., see the top solutions in the AMEX competition [206]). For more details on all preprocessing steps, see Appendix C.2. Compute Resources. We used CPU/GPU virtual machines via GCP [207]; for specifications, see Appendix C.2. The estimated cost of BeyondArena was ∼$50k and took ∼16.25 wall-clock years.
6
Metrics. We chose metrics aligning with prior benchmarks [11, 15, 101] and machine learning literature [208–210]. We use ROC AUC for binary classification, log-loss for multiclass classification, and RMSE for regression. We chose ROC AUC because it is invariant to threshold tuning for binary classification, unlike accuracy as used by TALENT [12]. For multiclass classification, we use log-loss because it is a proper scoring rule [209, 211]. We use RMSE, a point prediction metric, for regression, and do not use probabilistic prediction metrics [23, 212, 213]. Point predictions are closer to the application’s objectives across almost all datasets we investigated. Thus, RMSE is more representative for practitioners. We normalized and aggregated the model performances with Elo [214] and mean Improvability [11], using implementations from TabArena. Improvability (0% − 100%) measures error relative to the best method on each outer split, making it sensitive to performance gaps. Elo ratings are computed by modeling pairwise comparisons and reflect the probability of winning. We always calibrate 1000 Elo to be the performance of a default XGBoost.
6
Results
BeyondArena aims to understand how well tabular foundation models would perform when practitioners use them in real-world predictive applications. Therefore, we first assess the extent to which tabular foundation models are truly general, and then investigate the validity of our results through ablation studies. To investigate the performance with respect to dataset characteristics, we define 12 sub-benchmarks: one per task type (IID, Grouped, Temporal); four for dataset scale (Tiny for datasets with 100 ≤ n < 1k training rows, Small for 1k ≤ n < 10k, Medium for 10k ≤ n < 100k, Large for 100k ≤ n); two for dataset dimensionality (Low for datasets with m ≤ 100 columns after preprocessing, High for m > 100); two for special feature types (Text for datasets with text features, High Card. for datasets with at least one categorical column that has more than 50 categories); and finally the full benchmark containing All datasets. Figure 4 shows the Elo and Improvability for all methods on all datasets. Figure 5 presents the Elo per sub-benchmark, and Figure 1 shows the Elo of the best model per family. When do tabular foundation models outperform traditional models? TFMs perform best on small, tiny, and IID datasets (Figure 1). Moreover, TFMs are also the best on average when ranked by Improvability, or when comparing default performance (Figure 4, Leaderboard as table in Appendix D.2). However, traditional approaches – particularly deep learning models such as RealMLP – benefit substantially from tuning and ensembling. Consequently, the in-context learning performance of all TFMs fails to compete with traditional models on non-IID (temporal and grouped), large-scale, high-dimensional, and high-cardinality datasets (Figure 5). In general, as sub-benchmarks are not independent (Section 4), bad performance on one (e.g., large) might also explain bad performance on another (e.g., temporal). Lastly, we observe that the default training costs, including cross-validation, of TFMs remain significantly lower than those of their tuned competitors (Figure 4, right), and continue to have large inference costs (Appendix D.1). We present results per dataset in Appendix G. To ground our findings in formal statistical evidence, we complement them with a suite of nonparametric tests in Appendix E. In general, our analysis shows that the global ranking and pairwise comparisons across models and sub-benchmarks are significant. In detail, we answer 5 concrete questions about our results, such as whether the methods really perform differently (§E.1) or which method a practitioner should pick for each data regime (§E.5). When are TFMs all you need for peak-performance? In Figure 6, we investigate the ability of TFMs to achieve peak performance (rank 1) on individual datasets, showing that (1) TabICLv2 ranks first on 19% of all datasets, followed by TabPFN-2.6 (10.5%). (2) TFMs significantly outperform other models on 49% of datasets, and at least one of the TFMs performs as well as the best non-TFM on an additional 21%, making TFMs a viable solution to get peak performance on 70% of all datasets. (3) On grouped, temporal, and large datasets, TFMs attain peak performance only on a small fraction of datasets. Overall, TFMs are clearly outperformed on 42 datasets, which are large, high-dimensional, non-IID, or include high-cardinality categorical features, illustrating the need for unified, holistic benchmarks to further improve TFMs across a wider variety of datasets. BeyondArena Ablations To better understand BeyondArena and inform future benchmarks, we ablate our experiment setup as described in Section 5. We summarize the results for the majority of our ablations in Table 1. 7
Default
Tuned
Tuned + Ensembled
1200
20
Improvability (%)
22
1100 1000 900
18 16 14 12 10
800
RandomForest ExtraTrees TabDPT XGBoost LightGBM CatBoost TabM RealMLP TabICLv2 TabPFN-2.6
Op tim al
Elo
Partially imputed 1300
101
Linear ExtraTrees XGBoost TabICLv2 TabM RealMLP RandomForest TabDPT LightGBM CatBoost TabPFN-2.6
102
103
Train time per 1K samples (s) (median)
Figure 4: BeyondArena Leaderboard (Left=Elo, Right=Improvability). We show the performance of 11 models across 142 datasets with the default hyperparameters, with tuning, and with tuning and post-hoc ensembling. The default performance of TFMs equals their in-context learning performance.
Elo
Family: Model:
Foundation Model
RealMLP
MLP
GBDT
Baseline
TabPFN-2.6*
TabM
CatBoost
1500 1400 Task 1300 1200 1100 1000 900 800 700 600
IID
N=103
Imputed for Large Datasets (>100k rows)
LightGBM
TabICLv2
Scale
Grouped N=18
Temporal N=21
Tiny
N=52
XGBoost
TabDPT*
Dimensionality
Small N=38
Medium N=30
Large N=22
Low
N=91
ExtraTrees
RandomForest
Linear
Features
High N=51
Text
N=16
High Card. N=30
All
N=142
Figure 5: Elo per model across sub-benchmarks. For each sub-benchmark of BeyondArena, we show the performance of each traditional model with tuning and post-hoc ensembling. For TFMs, we show their in-context learning performance. We aggregate results per model family in Figure 1.
Different Outer Splits. (A.1) We use up to 60 outer validation splits in BeyondArena, which drastically increases compute costs. To reduce environmental impact and enable compute-constrained researchers, we would like to use fewer outer splits. Erickson et al. [11] introduced TabArena-Lite, saving compute by using only the first split per dataset, which produces a comparable ranking on average. We instead introduce BeyondArena-Core, by automatically selecting a subset of splits per dataset, ensuring that the ranking per dataset on the subset is comparable to that obtained with all splits. Compared to BeyondArena, -Lite would be ×9 and -Core ×5 faster. Yet, the per dataset win rates of -Core are ×2.3 more stable than -Lite; see Appendix F.1 for details. (A.2) Besides using fewer outer splits, we ablate using IID splits for non-IID data. Prior work has shown that inappropriate splits confound the evaluation of temporal data, cf. [15, 75]. In Appendix F.2, we show that, for two exemplary grouped datasets, inappropriate outer splits can drastically distort model rankings (τ = 0.49 and τ = 0.60) and raw performance estimates. Hence, it is pivotal to use appropriate non-IID splits when evaluating grouped data. Different Inner Splits. (B.1) BeyondArena uses 5-repeated 5-fold inner cross-validation (5 × 5 CV) for datasets with fewer than 500 training samples, and 8-fold CV otherwise, to avoid overtuning hurting test performance [198, 199]. Table 1 and Appendix F.3 show that 5 × 5 CV consistently achieves a higher test performance, verifying the correctness of our setup. (B.2) We use non-IID inner splits for non-IID tasks. Table 1 shows that using IID inner splits results in similar ranks, but non-IID splits perform better on average and can reach significantly higher peaks (see Appendix F.4). Thus, using non-IID inner splits for temporal and grouped data was appropriate. Different Preprocessing. We use preprocessing for grouped data to handle the group index column. We ablate using only the raw data in Table 1 and Appendix F.5. (C.1a) For L-P-G data, the model rankings are mostly stable, while (C.1b) the rank ordering is mixed up for L-P-S data. In both cases, BeyondArena’s preprocessing is better on average, yet TabPFN-2.6 improves substantially when preprocessing is disabled (F.5). The results invite future work. For BeyondArena, we conclude that the performance of most models is representative, whereas TabPFN-2.6 is negatively biased. (C.2.a/b) We used Qwen3-Embedding-8B to encode text features. We compare Qwen3 to TF-IDF from skrub [201], a classical text encoder in Table 1 and Appendix F.6. For short-text datasets, Qwen3 8
Table 1: Ablation Results. We show Kendall’s tau (τ ) [215] and the win rate when ablating our experimental setup. τ ∈ [−1, 1] measures the difference in ranking, with 1.0 being identical ranks. Win rate shows how often BeyondArena’s setting leads to a higher average rank across models. We split ablations for grouped data: label-per-group (L-P-G) / label-per-sample (L-P-S); and text data: ≤ 50 characters on average (short) / > 50 (long). We share details in Appendix F.
Figure 6: Rank-1 share across subbenchmarks. We show the share of datasets where each model (family) significantly outperforms the others. Significance is measured by a Wilcoxon signed-rank test with α = 0.05 for datasets with >3 outer splits and by whether a model wins all splits otherwise. MLP, GBDT, win for the family; TFM, win for the family, no winner within the family; non-TFM no clear winner for the best GBDT and MLP; Mixed no clear winner for the best TFM and non-TFM.
Part
Ablation
Setting (BeyondArena → new)
Kendall τ
Win Rate
Inner Splits
(B.1) (B.2)
5×5-fold CV → 8-fold CV Non-IID → IID
0.93 1.00
100% 100%
Grouped Data Pre-processing
(C.1a) (C.1b)
(L-P-G) Agg. Index → N/A (L-P-S) Drop Index → N/A
0.81 0.43
71% 71%
Text Data Pre-processing
(C.2a) (C.2b)
(Short Text) Qwen3 → TF-IDF (Long Text) Qwen3 → TF-IDF
1.00 0.89
78% 6%
Post-processing
(D)
N/A → Probability Calibration
0.85
18%
performs better than TF-IDF. For long texts, TF-IDF generally yields better predictive performance. Yet, the model rankings are only slightly affected; thus, our results are still representative. In the future, we recommend including preprocessing in the search space for grouped and text data. Probability Calibration for Log-loss. (D) We use log-loss as a metric for multiclass classification. Log-loss is affected by probability calibration, and thus, post hoc calibration methods can often improve performance. BeyondArena does not use post hoc calibration by default; we ablate adding it based on the saved predictions. In Table 1, we show that calibration would have performed better most of the time. Across all models, only TabPFN-2.6 and RealMLP perform worse with calibration (Appendix F.7). Thus, we recommend using calibration by default for other models going forward.
7
Conclusion
We introduced BeyondArena, a unified benchmark for predictive modeling on tabular data that consolidates diverse task types (IID, temporal, grouped), dataset scales, feature dimensionality, and feature types. Together with DataFoundry, a framework for reproducible dataset curation, we enable standardized evaluation across 142 datasets and 11 models. Our results show that tabular foundation models (TFMs) perform well on small- to medium-scale IID data, but are outperformed by traditional methods in non-IID, large-scale, and high-dimensional settings, highlighting key gaps for future research. In summary, TFMs generalize well to existing paradigms and show very promising in-context learning performance, but they are not yet fully general for tabular data. Limitations and Societal Impact. The conclusions of BeyondArena are limited by: (1) We tune models for 25 random configurations due to compute constraints. (2) We evaluate in-context learning performance of TFMs, but do not investigate fine-tuning TFMs. (3) We include the top three TFMs from TabArena, but do not explore the performance of other open- or closed-source models. (4) Across the 142 datasets, some sub-benchmarks have more datasets than others, leading to sub-benchmarks being under- and over-representative in the leaderboard computed across all datasets. (5) We make the first step towards a unified pipeline for preprocessing, validation, and tuning across task types and feature types. Yet, the best practices for validation protocols for temporal and grouped data, as well as for preprocessing grouped and text data, remain to be explored. On the societal side, BeyondArena improves rigor and reproducibility, narrowing the gap between academics and practitioners at non-trivial computational cost – which we trade off using BeyondArena-Core. Future Work. To broaden the scope of evaluations of TFMs and further explore their generalizability, future directions could extend BeyondArena to few-shot predictions, more multimodal tabular data (e.g., tabular-image-text), and other tasks such as relational learning or survival analysis. To conclude, BeyondArena surfaces critical limitations of current TFMs and establishes a new evaluation standard for tabular machine learning on broader, more demanding challenges. 9
Acknowledgments and Disclosure of Funding L.P. acknowledges funding by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under SFB 1597 (SmallData), grant number 499552394. A.T. acknowledges funding by the German Federal Ministry for Economic Affairs and Energy (BMWE). A.A. is supported by an Israel Ministry of Science and Technology (MOST) grant on multi-modal AI. A.P. was funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Commission. Neither the European Union nor the European Commission can be held responsible for them. This work was supported by the European Union’s Horizon Europe research and innovation programme under grant agreement No 101214398 (ELLIOT).
M.T. is supported by the Konrad Zuse School of Excellence in Learning and Intelligent Systems (https://eliza.school/) through the DAAD programme Konrad Zuse Schools of Excellence in Artificial Intelligence, sponsored by the Federal Ministry of Education and Research. G.V. acknowledges support from ANR via grant TaFoMo (ANR-25-CE23-1822). This work is partly supported by Hi! PARIS and ANR/France 2030 program (ANR-23-IACL-0005). F.H. acknowledges the financial support of the Hector Foundation.
Author Contributions L.P. led the code development, led the experiments, led the writing of the paper, led the evaluation and visualizations, led dataset curation, and led the management of the collaboration. A.T. contributed to the writing of the paper, contributed to evaluation and visualizations, and contributed to dataset curation. N.E. contributed to the writing of the paper and contributed to the evaluation and visualizations. G.B. helped with writing the paper, contributed to the evaluation and visualizations, helped with dataset curation, and helped with reviewing and editing. D.H. contributed to writing the paper, helped with the management of the collaboration, and helped with supervising the project. A.A. helped with writing the paper, and contributed to reviewing and editing. A.P. helped with dataset curation. M.T. helped with dataset curation. G.V. helped with evaluation and visualizations, helped with management of the collaboration, and supervised G.B. during the project. F.H. supervised L.P. during the project.
Competing Interests D.H. is one of the authors of RealMLP. L.P. and F.H. are a subset of the authors of TabPFN-2.6. D.H. and G.V. are a subset of the authors of TabICLv2. L.P., A.T., N.E., and D.H. are the maintainers of TabArena. L.P. and N.E. are maintainers of AutoGluon. G.V. is a core contributor to skrub. G.V. is (co)founder of scikit-learn. L.P., N.E., A.A., and F.H. are affiliated with Prior Labs, a company focused on developing tabular foundation models. G.V. is affiliated with probabl, a company focused on developing software for industrial-grade data science. The authors declare no other competing interests.
References [1] Jeongwhan Choi, Woosung Kang, Minseo Kim, Jongwoo Kim, and Noseong Park. Can tabpfn compete with gnns for node classification via graph tabularization? arXiv preprint arXiv:2512.08798, 2025. [2] Dmitry Eremeev, Gleb Bazhenov, Oleg Platonov, Artem Babenko, and Liudmila Prokhorenkova. Turning tabular foundation models into graph foundation models. arXiv preprint arXiv:2508.20906, 2025. 10
[3] Adrian Hayler, Xingyue Huang, Ismail Ilkan Ceylan, Michael Bronstein, and Ben Finkelshtein. Bringing graphs to the table: Zero-shot node classification via tabular foundation models. arXiv preprint arXiv:2509.07143, 2025. [4] Tianyin Liao, Chunyu Hu, Yicheng Sui, Xingxuan Zhang, Peng Cui, Jianxin Li, and Ziwei Zhang. Tfmlinker: Universal link predictor by graph in-context learning with tabular foundation models. arXiv preprint arXiv:2602.08592, 2026. [5] Rosen Ting-Ying Yu, Cyril Picard, and Faez Ahmed. Git-bo: High-dimensional bayesian optimization with tabular foundation models. arXiv preprint arXiv:2505.20685, 2025. [6] Yukun Du, Haiyue Yu, Xiaotong Xie, Yan Zheng, Lixin Zhan, Yudong Du, Chongshuang Hu, Boxuan Wang, and Jiang Jiang. Meta-black-box optimization with bi-space landscape analysis and dual-control mechanism for saea. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 36891–36899, 2026. [7] Jeffrey Hu, Rongzhi Dong, Ying Feng, Ming Hu, and Jianjun Hu. Foundation-model surrogates enable data-efficient active learning for materials discovery, 2026. [8] Pablo García, J de Curtò, I de Zarzà, Juan Carlos Cano, and Carlos T Calafate. Foundation models for cybersecurity: A comprehensive multi-modal evaluation of tabpfn and tabicl for tabular intrusion detection. Electronics, 14(19):3792, 2025. [9] Viacheslav Barkov, Jonas Schmidinger, Robin Gebbers, and Martin Atzmueller. Modern neural networks for small tabular datasets: The new default for field-scale digital soil mapping? European Journal of Soil Science, 77(2):e70299, 2026. [10] Lawrence A Shaktah, Marco Gustav, Tim Lenz, Junhao Liang, Lars Hilgers, Zunamys I Carrero, and Jakob Nikolas Kather. Established machine learning matches tabular foundation models in clinical predictions. medRxiv, pages 2026–02, 2026. [11] Nick Erickson, Lennart Purucker, Andrej Tschalzev, David Holzmüller, Prateek Mutalik Desai, David Salinas, and Frank Hutter. Tabarena: A living benchmark for machine learning on tabular data. In Proceedings of the 39th Conference on Neural Information Processing Systems (NeurIPS), 2025. [12] Si-Yang Liu, Hao-Run Cai, Qi-Le Zhou, and Han-Jia Ye. Talent: A tabular analytics and learning toolbox. arXiv preprint arXiv:2407.04057, 2024. [13] Xingxuan Zhang, Gang Ren, Han Yu, Hao Yuan, Hui Wang, Jiansheng Li, Jiayun Wu, Lang Mo, Li Mao, Mingchao Hao, et al. Limix: Unleashing structured-data modeling capability for generalist intelligence. arXiv preprint arXiv:2509.03505, 2025. [14] Ricardo Knauer, Marvin Grimm, and Erik Rodner. Pmlbmini: A tabular classification benchmark suite for data-scarce applications. In AutoML Conference 2024 (ABCD Track), 2024. [15] Ivan Rubachev, Nikolay Kartashev, Yury Gorishniy, and Artem Babenko. Tabred: Analyzing pitfalls and filling the gaps in tabular deep learning benchmarks. arXiv preprint arXiv:2406.19380, 2024. [16] Martin Mráz, Breenda Das, Anshul Gupta, Lennart Purucker, and Frank Hutter. Towards benchmarking foundation models for tabular data with text. arXiv preprint arXiv:2507.07829, 2025. [17] Han-Jia Ye, Si-Yang Liu, and Wei-Lun Chao. A closer look at tabpfn v2: Strength, limitation, and extension. arXiv preprint arXiv:2502.17361, 2025. [18] Frederik Hoppe, Lars Kleinemeier, Astrid Franz, and Udo Göbel. Comparing task-agnostic embedding models for tabular data. arXiv preprint arXiv:2511.14276, 2026. [19] Zi-Jian Cheng, Zi-Yi Jia, Zhi Zhou, Yu-Feng Li, and Lan-Zhe Guo. Realistic evaluation of tabpfn v2 in open environments. arXiv preprint arXiv:2505.16226, 2025. [20] Sam Schiffman. Do foundation models learn fair representations? a critical evaluation of tabpfn on algorithmic fairness benchmarks. TRUST-AI: The European Workshop on Trustworthy AI., 2025. [21] Shi Bin Hoo, Samuel Müller, David Salinas, and Frank Hutter. The tabular foundation model tabpfn outperforms specialized time series forecasting models based on simple features. In NeurIPS 2024 Third Table Representation Learning Workshop, 2024. 11
[22] Linjie Xu, Yanlin Zhang, Quan Gan, Minjie Wang, and David Wipf. No need to train your rdb foundation model. arXiv preprint arXiv:2602.13697, 2026. [23] Jonas Landsgesell, Pascal Knoll, and Tizian Wenzel. Distributional regression with tabular foundation models: Evaluating probabilistic predictions via proper scoring rules, 2026. URL https://arxiv.org/abs/2603.08206. [24] Qiong Zhang, Yan Shuo Tan, Qinglong Tian, and Pengfei Li. Tabpfn: One model to rule them all? arXiv preprint arXiv:2505.20003, 2025. [25] Jessup Byun, Xiaofeng Lin, Joshua Ward, and Guang Cheng. Risk in context: Benchmarking privacy leakage of foundation models in synthetic tabular data generation. arXiv preprint arXiv:2507.17066, 2025. [26] Nguyen Gia Hien Vu, Yifan Tang, Rey Lim, Yifan Yang, Hang Ma, Ke Wang, and G Gary Wang. Adaptation and fine-tuning with tabpfn for travelling salesman problem. arXiv preprint arXiv:2511.05872, 2025. [27] Afonso Lourenço, João Gama, Eric P Xing, and Goreti Marreiros. Bridging streaming continual learning via in-context large tabular models. arXiv preprint arXiv:2512.11668, 2025. [28] Lars Henry Berge Olsen and Dennis Christensen. Computing conditional shapley values using tabular foundation models. arXiv e-prints, pages arXiv–2602, 2026. [29] Rosen Ting-Ying Yu, Nicholas Sung, and Faez Ahmed. Fire: Multi-fidelity regression with distribution-conditioned in-context learning using tabular foundation models. arXiv preprint arXiv:2601.22371, 2026. [30] Jianqiao Zheng, Cameron Gordon, Yiping Ji, Hemanth Saratchandran, and Simon Lucey. From tables to signals: Revealing spectral adaptivity in tabpfn. arXiv preprint arXiv:2511.18278, 2025. [31] David Schiff, Ofir Lindenbaum, and Yonathan Efroni. Gradient free deep reinforcement learning with tabpfn. arXiv preprint arXiv:2509.11259, 2025. [32] Yunhui Liu, Tieke He, Yongchao Liu, Can Yi, Hong Jin, and Chuntao Hong. Tabular foundation models are strong graph anomaly detectors. arXiv preprint arXiv:2601.17301, 2026. [33] Da In Kim, Wei Siang Lai, and Kelly W Zhang. Tabular foundation models can do survival analysis. arXiv preprint arXiv:2601.22259, 2026. [34] Erkan Karabulut, Daniel Daza, Paul Groth, Martijn C Schut, and Victoria Degeler. Tabular foundation models can learn association rules. arXiv preprint arXiv:2602.14622, 2026. [35] Patryk Marszałek, Tomasz Kuśmierczyk, and Marek Śmieja. Tactic for navigating the unknown: Tabular anomaly detection via in-context inference. arXiv preprint arXiv:2603.14171, 2026. [36] Feiyu Pan, Tianbin Zhang, Aoqian Zhang, Yu Sun, Zheng Wang, Lixing Chen, Li Pan, and Jianhua Li. Lakemlb: Data lake machine learning benchmark. arXiv preprint arXiv:2602.10441, 2026. [37] Daniiar Dyikanov, Aleksandr Zaitsev, Tatiana Vasileva, Iris Wang, Arseniy A Sokolov, Evgenii S Bolshakov, Alena Frank, Polina Turova, Olga Golubeva, Anna Gantseva, et al. Comprehensive peripheral blood immunoprofiling reveals five immunotypes with immunotherapy response characteristics in patients with cancer. Cancer cell, 42(5):759–779, 2024. [38] Vinh Quang Tran and Haewon Byeon. Predicting dementia in parkinson’s disease on a small tabular dataset using hybrid lightgbm–tabpfn and shap. Digital Health, 10: 20552076241272585, 2024. [39] Vinh Nguyen Dao, Nhat-Thang Tran, Ta-Son Vo, Hong-Thinh Le, Thu-Ha Thi Nguyen, Quoc-Huy Vu Nguyen, Minh-Thi Thi Ha, Tam Minh Le, Diem-Tuyet Thi Hoang, KhanhTrang Nguyen Huynh, et al. Early prediction of gestational diabetes using integrated cell-free dna features and omics-derived genetic scores. medRxiv, pages 2025–09, 2025. [40] Mert Karabacak, Burak Berksu Ozkara, Tobias D Faizy, Trevor Hardigan, Jeremy J Heit, Dhairya A Lakhani, Konstantinos Margetis, J Mocco, Kambiz Nael, Max Wintermark, et al. Data-driven prognostication in distal medium vessel occlusions using explainable machine learning. American Journal of Neuroradiology, 46(4):725–732, 2025. 12
[41] Luis Magadán, José Roldán-Gómez, Juan Carlos Granda, and Francisco José Suárez. Early fault classification in rotating machinery with limited data using tabpfn. IEEE Sensors Journal, 23(24):30960–30970, 2023. [42] Sarah A Alzakari, Asma Aldrees, Muhammad Umer, Lucia Cascone, Nisreen Innab, and Imran Ashraf. Artificial intelligence-driven predictive framework for early detection of still birth. SLAS technology, 29(6):100203, 2024. [43] Ryunosuke Noda, Daisuke Ichikawa, and Yugo Shibagaki. Machine learning-based diagnostic prediction of minimal change disease: model development study. Scientific reports, 14(1): 23460, 2024. [44] Hyunseok Yang and Jungsu Park. Comparing the performance of a deep learning model (tabpfn) for predicting river algal blooms with varying data composition. Journal of Wetlands Research, 26(3):197–203, 2024. [45] Peng Wang, Hongjun Liu, Yiming Shi, Ao Liu, Qingyu Zhu, Irina Albu, Maja Pacholec, Lulu Cheng, Xu Sun, and Xinli Chi. Harnessing small-data machine learning for transformative mental health forecasting: Towards precision psychiatry with personalised digital phenotyping, 2025. [46] Yunhua Li, Jianfeng Yang, Pan Xiao, Haibo Liu, Yingjun Zhou, Xiuqi Yang, Gangwen Chen, and Zhichao Zuo. Mri delta-radiomics and morphological feature-driven tabpfn model for preoperative prediction of lymphovascular invasion in invasive breast cancer. Technology in Cancer Research & Treatment, 24:15330338251362050, 2025. [47] Xun Li and Yujing Jiang. A tabpfn-based framework for slope stability analysis using geometric features and shear strength parameters. Rock Mechanics Bulletin, page 100326, 2026. [48] Bai Liu, Yun Chen, and Dazhi Yang. Evaluating tabpfn for regression tasks in solar energy meteorology. Solar Energy, 309:114472, 2026. [49] Giulia Perciballi, Federica Granese, Ahmad Fall, Farida Zehraoui, Edi Prifti, and Jean-Daniel Zucker. Adapting tabpfn for zero-inflated metagenomic data. In NeurIPS 2024 Third Table Representation Learning Workshop, 2024. [50] Enoch Chi Ngai Lim and Chi Eung Danforn Lim. Redefining surgical health economics: The potential of tabpfn for real-time precision modelling. Journal of Computer and Communications, 13(11):30–40, 2025. [51] Summer Zhou, Vinayak Agarwal, Ashwin Gopinath, and Timothy Kassis. The limitations of tabpfn for high-dimensional rna-seq analysis. bioRxiv, pages 2025–08, 2025. [52] Jonas Schmidinger, Viacheslav Barkov, Sebastian Vogel, Martin Atzmueller, and Gerard BM Heuvelink. Kriging prior regression: A case for kriging-based spatial features with tabpfn in soil mapping. Computers and Electronics in Agriculture, 243:111352, 2026. [53] F Schwarz, L Levien, M Maulhardt, G Wulf, N Brökers, and E Aydilek. Predicting adverse events for risk stratification of chemotherapy based stem cell mobilization in multiple myeloma. npj Digital Medicine, 2026. [54] Wenpeng Zhao, Shanchuan Guo, Xueliang Zhang, Pengfei Tang, Xiaoquan Pan, Haowei Mu, Chenghan Yang, Zilong Xia, Zheng Wang, Jun Du, et al. A weakly supervised approach for large-scale agricultural parcel extraction from vhr imagery via foundation models and adaptive noise correction. ISPRS Journal of Photogrammetry and Remote Sensing, 233:180–208, 2026. [55] Chris Varghese, Elizabeth Habermann, Kristine Hanson, Ashok Choudhary, Hojjat Salehinejad, and Cornelius Thiels. Tabular foundation models as a new portable standard in local surgical risk prediction. Surgery, 192:110078, 2026. [56] Yilang Ding, Jiawen Ren, Jiaying Lu, Gloria Hyunjung Kwak, Armin Iraji, Shengpu Tang, and Alex Fedorov. Longitudinal progression prediction of alzheimer’s disease with tabular foundation model. arXiv preprint arXiv:2508.17649, 2025. [57] Hualiang Zhu, Xianwei Zhang, Gang Wei, Qingzhi Wang, Xinyu Liu, Lei Yan, and Gang Wang. A method for better mapping of susceptibility to thaw hazards in data-scarce cold regions. Remote Sensing of Environment, 337:115338, 2026. [58] Carola Sophia Heinzel, Lennart Purucker, Frank Hutter, and Peter Pfaffelhuber. Advancing biogeographical ancestry predictions through machine learning. Forensic Science International: Genetics, 79:103290, 2025. 13
[59] Eric Johnsson, Shrinjay Sharma, Arvind Gangoli Rao, David Dubbeldam, Sofia Calero, and Thijs JH Vlugt. Predicting the maximum loading in zeolites for hydroisomerization applications: A machine learning approach. The Journal of Physical Chemistry C, 2026. [60] Jian Guo, Haoxuan Ren, and Kaijiang Ma. Predicting initial accident states in hazardous chemical road transportation: a causal and interpretable machine learning approach. Reliability Engineering & System Safety, page 112430, 2026. [61] Gil Hwan Wang, Sujith Mangalathu, and Jong-Su Jeon. Transformer-based foundation models for assessing earthquake-and vehicle-induced damage in bridges. Engineering Structures, 358: 122587, 2026. [62] Bichen Shang, Guo Li, Weijie Sun, Liwei Zhang, Guanzhe Cui, Jiyuan Tu, Xiang Fang, and Xueren Li. In-context learning for nano-pcm thermal behavior prediction in battery thermal management via lattice boltzmann simulation. Energy, page 138693, 2025. [63] Dan-Ni Wu, Joey Jen, Erickson Fajiculay, Min-Fen Hsu, Ming-Chu Chang, Jen-Chen Yeh, Karen Sargsyan, Juozas Kupcinskas, Jurgita Skieceviciene, Ruta Steponaitiene, et al. Panmetaia high performance tabular foundation model for accurate pancreatic cancer diagnosis via nmr metabolomics. Nature Communications, 17(1):1595, 2026. [64] Giuseppe Romano, Pietro Bilancia, Alberto Locatelli, Mirko Mucciarini, Manuel Iori, and Marcello Pellicciari. A machine learning–based tool for enhancing position accuracy in industrial robots with a reduced dataset. Robotics and Computer-Integrated Manufacturing, 101:103289, 2026. [65] Muhammad Moshiur Rahman, Andrew Robson, and Theo Bekker. Machine learning approaches for assessing avocado alternate bearing using sentinel-2 and climate variables—a case study in limpopo, south africa. Remote Sensing, 17(24):3935, 2025. [66] Gang Chen, Zihan Yang, Peng Sun, Junfeng Li, Chenglong Wang, Jinliang Li, Guang Yang, and Likun Pan. Data-augmented machine learning for predicting biomass-derived hard carbon anode performance in sodium-ion batteries. Journal of Energy Chemistry, 2026. [67] Hongjian Chen, Feifei Fang, Pujun Long, Putian Yang, and Wei Guo. Coupling eur prediction with fracturing optimization: An integrated machine learning framework for shale gas development. Unconventional Resources, page 100246, 2025. [68] Huixia Zhang, Jiajun Tong, Minmin Chen, and Xichuan Cao. Boosting pre-trained model with silica nanoparticles cellular toxicity prediction. Scientific Reports, 2025. [69] Junjie Xu, Yilei Yu, Lihu Yang, Xin Wei, Shiqin Wang, Bingxia Liu, and Yamin Shi. Multiscale prediction from ion concentrations to salinity patterns of arid and saline-alkali farmland using machine learning algorithm in xinjiang region of china. Available at SSRN 5591702, 2025. [70] Nicolò Bellarmino, Riccardo Cantoro, Martin Huch, Tobias Kilian, and Annachiara Ruospo. Minimal supervision, maximum accuracy: Tabpfn for microcontroller performance prediction. In 2025 IEEE International Test Conference (ITC), pages 470–473. IEEE, 2025. [71] Justus Viga, Penelope Mueck, Alexander Löser, and Torben Weis. Fuelcast: Benchmarking tabular and temporal models for ship fuel consumption. In International Workshop on Advanced Analytics and Learning on Temporal Data, pages 54–69. Springer, 2025. [72] Jasmin ZK Chu, Joel CM Than, and Hudyjaya Siswoyo Jo. Deep learning for cross-selling health insurance classification. In 2024 International Conference on Green Energy, Computing and Sustainable Technology (GECOST), pages 453–457. IEEE, 2024. [73] Taiga Saito, Yu Otake, and Stephen Wu. Tabular foundation model for geoai benchmark problems bm/airportsoilproperties/2/2025. arXiv preprint arXiv:2509.03191, 2025. [74] Amit Lal. Evaluating sap rpt-1 for enterprise business process prediction: In-context learning vs. traditional machine learning on structured sap data. arXiv preprint arXiv:2602.19237, 2026. [75] Andrej Tschalzev, Lennart Purucker, Stefan Lüdtke, Frank Hutter, Christian Bartelt, and Heiner Stuckenschmidt. Unreflected use of tabular data repositories can undermine research quality. In The Future of Machine Learning Data Practices and Repositories at ICLR 2025, 2025. [76] Chris Chatfield. Time-series forecasting. Chapman and Hall/CRC, 2000. 14
[77] Bryan Lim and Stefan Zohren. Time-series forecasting with deep learning: a survey. Philosophical transactions of the royal society a: mathematical, physical and engineering sciences, 379(2194), 2021. [78] Oleksandr Shchur, Ali Caner Turkmen, Nick Erickson, Huibin Shen, Alexander Shirkov, Tony Hu, and Bernie Wang. Autogluon–timeseries: Automl for probabilistic time series forecasting. In International Conference on Automated Machine Learning, pages 9–1. PMLR, 2023. [79] Mayuka Jayawardhana, Nihal Sharma, Kazem Meidani, Bayan Bruss, Tom Goldstein, and Doron Bergman. Zero-shot multivariate time series forecasting using tabular prior fitted networks. arXiv preprint arXiv:2604.08400, 2026. [80] Andres Potapczynski, Ravi Kiran Selvam, Tatiana Konstantinova, Malcolm Wolff, Kin G Olivares, Ruijun Ma, Michael W Mahoney, Andrew Gordon Wilson, Boris N Oreshkin, and Dmitry Efimov. The arrow of time: What tabular foundation models miss in time series forecasting. In 1st ICLR Workshop on Time Series in the Age of Large Models, 2026. [81] Taha Aksu, Gerald Woo, Juncheng Liu, Xu Liu, Chenghao Liu, Silvio Savarese, Caiming Xiong, and Doyen Sahoo. Gift-eval: A benchmark for general time series forecasting model evaluation. arXiv preprint arXiv:2410.10393, 2024. [82] Oleksandr Shchur, Abdul Fatir Ansari, Caner Turkmen, Lorenzo Stella, Nick Erickson, Pablo Guerron, Michael Bohlke-Schneider, and Yuyang Wang. fev-bench: A realistic benchmark for time series forecasting. arXiv preprint arXiv:2509.26468, 2025. [83] Azul Garza, Renée Rosillo, Rodrigo Mendoza-Smith, David Salinas, Andrew Robert Williams, Arjun Ashok, Mononito Goswami, and José Martín Juárez. Impermanent: A live benchmark for temporal generalization in time series forecasting. arXiv preprint arXiv:2603.08707, 2026. [84] Zhongzheng Qiao, Sheng Pan, Anni Wang, Viktoriya Zhukova, Yong Liu, Xudong Jiang, Qingsong Wen, Mingsheng Long, Ming Jin, and Chenghao Liu. It’s time: Towards the next generation of time series forecasting benchmarks. arXiv preprint arXiv:2602.12147, 2026. [85] David Holzmüller, Léo Grinsztajn, and Ingo Steinwart. Better by default: Strong pre-tuned mlps and boosted trees on tabular data. Advances in Neural Information Processing Systems, 37:26577–26658, 2024. [86] Yury Gorishniy, Ivan Rubachev, Nikolay Kartashev, Daniil Shlenskii, Akim Kotelnikov, and Artem Babenko. Tabr: Tabular deep learning meets nearest neighbors. In The Twelfth International Conference on Learning Representations, 2024. [87] Yury Gorishniy, Akim Kotelnikov, and Artem Babenko. Tabm: Advancing tabular deep learning with parameter-efficient ensembling. arXiv preprint arXiv:2410.24210, 2024. [88] N. Hollmann, S. Müller, L. Purucker, A. Krishnakumar, M. Körfer, Shi Bin Hoo, Robin Tibor Schirrmeister, and Frank Hutter. Accurate predictions on small data with a tabular foundation model. Nature, 637(8045):319–326, 2025. [89] Y. Gorishniy, I. Rubachev, V. Khrulkov, and A. Babenko. Revisiting deep learning models for tabular data. In M. Ranzato, A. Beygelzimer, K. Nguyen, P. Liang, J. Vaughan, and Y. Dauphin, editors, Proceedings of the 34th International Conference on Advances in Neural Information Processing Systems (NeurIPS’21), pages 18932–18943. Curran Associates, 2021. [90] Ravid Shwartz-Ziv and Amitai Armon. Tabular data: Deep learning is not all you need. Information Fusion, 81:84–90, 2022. [91] L. Grinsztajn, E. Oyallon, and G. Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Proceedings of the 35th International Conference on Advances in Neural Information Processing Systems (NeurIPS’22). Curran Associates, 2022. [92] D. McElfresh, S. Khandagale, J. Valverde, V. Prasad C, G. Ramakrishnan, M. Goldblum, and C. White. When do neural nets outperform boosted trees on tabular data? In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Proceedings of the 36th International Conference on Advances in Neural Information Processing Systems (NeurIPS’23), pages 76336–76369. Curran Associates, 2023. [93] Assaf Shmuel, Oren Glickman, and Teddy Lazebnik. A comprehensive benchmark of machine and deep learning across diverse tabular datasets. arXiv preprint arXiv:2408.14817, 2024. 15
[94] Guri Zabërgja, Arlind Kadra, Christian MM Frey, and Josif Grabocka. Tabular data: Is deep learning all you need? arXiv preprint arXiv:2402.03970, 2024. [95] Kyungeun Lee, Moonjung Eo, Hye-Seung Cho, Dongmin Kim, Ye Seul Sim, Seoyoon Kim, Min-Kook Suh, and Woohyung Lim. Multitab: A comprehensive benchmark suite for multidimensional evaluation in tabular domains. arXiv preprint arXiv:2505.14312, 2025. [96] Dihong Jiang, Ruoqi Cao, Zhiyuan Dang, Li Huang, Qingsong Zhang, Zhiyu Wang, Shihao Piao, Shenggao Zhu, Jianlong Chang, Zhouchen Lin, et al. Omnitabbench: Mapping the empirical frontiers of gbdts, neural networks, and foundation models for tabular data at scale. arXiv preprint arXiv:2604.06814, 2026. [97] Marc Hanussek, Matthias Blohm, and Maximilien Kintz. Can automl outperform humans? an evaluation on popular openml datasets using automl benchmark. In 2020 2nd international conference on artificial intelligence, robotics and control, pages 29–32, 2020. [98] Marc-André Zöller and Marco F Huber. Benchmark and survey of automated machine learning frameworks. Journal of artificial intelligence research, 70:409–472, 2021. [99] L. Purucker and J. Beel. Assembled-openml: Creating efficient benchmarks for ensembles in automl with openml. In I. Guyon, M. Lindauer, M. van der Schaar, F. Hutter, and R. Garnett, editors, First International Conference on Automated Machine Learning - Workshop Track, 2022. [100] Jiawei Jiang, Yi Wei, Yu Liu, Wentao Wu, Chuang Hu, Zhigao Zheng, Ziyi Zhang, Yingxia Shao, and Ce Zhang. How good are machine learning clouds? benchmarking two snapshots over 5 years. The VLDB Journal, 33(3):833–857, 2024. [101] P. Gijsbers, M. Bueno, S. Coors, E. LeDell, S. Poirier, J. Thomas, B. Bischl, and J. Vanschoren. Amlb: an automl benchmark. Journal of Machine Learning Research, 25(101):1–65, 2024. [102] Israel Campero Jurado, Pieter Gijsbers, and Joaquin Vanschoren. Automl benchmark with shorter time constraints and early stopping. In The Future of Machine Learning Data Practices and Repositories at ICLR 2025, 2025. [103] B. Bischl, G. Casalicchio, M. Feurer, F. Hutter, M. Lang, R. Mantovani, J. N. van Rijn, and J. Vanschoren. Openml benchmarking suites and the openml100. arXiv:1708.03731v1 [stat.ML], 2019. [104] B. Bischl, G. Casalicchio, M. Feurer, F. Hutter, M. Lang, R. Mantovani, J. van Rijn, and J. Vanschoren. OpenML benchmarking suites. In J. Vanschoren and S. Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks. Curran Associates, 2021. [105] R. Olson, W. La Cava, P. Orzechowski, R. Urbanowicz, and J. Moore. PMLB: a large benchmark suite for machine learning evaluation and comparison. BioData mining, 10:1–13, 2017. [106] Joseph D Romano, Trang T Le, William La Cava, John T Gregg, Daniel J Goldberg, Praneel Chakraborty, Natasha L Ray, Daniel Himmelstein, Weixuan Fu, and Jason H Moore. Pmlb v1.0: an open-source dataset collection for benchmarking machine learning methods. Bioinformatics, 38(3):878–880, 2022. [107] S. Fischer, L. Harutyunyan, M. Feurer, and B. Bischl. Openml-ctr23 – a curated tabular regression benchmarking suite. In A. Faust, C. White, F. Hutter, R. Garnett, and J. Gardner, editors, Second International Conference on Automated Machine Learning - Workshop Track, 2023. [108] David Salinas and Nick Erickson. Tabrepo: A large scale repository of tabular model evaluations and its automl applications. In AutoML Conference 2024 (ABCD Track), 2024. [109] Andrej Tschalzev, Sascha Marton, Stefan Lüdtke, Christian Bartelt, and Heiner Stuckenschmidt. A data-centric perspective on evaluating machine learning models for tabular data. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. [110] R. Kohli, M. Feurer, B. Bischl, K. Eggensperger, and F. Hutter. Towards quantifying the effect of datasets for benchmarking: A look at tabular machine learning. In Data-centric Machine Learning Research Workshop at the International Conference on Learning Representations, 2024. 16
[111] Kyungeun Lee, Moonjung Eo, Hye-Seung Cho, Min-Kook Suh, Seoyoon Kim, Ye Seul Sim, Suhee Yoon, Sanghyu Yoon, and Woohyung Lim. Range-limited augmentation for few-shot learning in tabular data with comprehensive benchmark. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pages 1262–1273, 2025. [112] Han-Jia Ye, Si-Yang Liu, Hao-Run Cai, Qi-Le Zhou, and De-Chuan Zhan. A closer look at deep learning methods on tabular datasets, 2025. URL https://arxiv.org/abs/2407.00956. [113] X. Shi, J. Mueller, N. Erickson, M. Li, and A. Smola. Benchmarking multimodal automl for tabular data with text fields. In J. Vanschoren and S. Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks. Curran Associates, 2021. [114] Léo Grinsztajn, Edouard Oyallon, Myung Jun Kim, and Gaël Varoquaux. Vectorizing string entries for data processing on tables: when are larger language models better? arXiv preprint arXiv:2312.09634, 2023. [115] Jiaying Lu, Yongchen Qian, Shifan Zhao, Yuanzhe Xi, and Carl Yang. Mug: A multimodal classification benchmark on game data with tabular, textual, and visual fields. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5332–5346, 2023. [116] Zhiqiang Tang, Haoyang Fang, Su Zhou, Taojiannan Yang, Zihan Zhong, Tony Hu, Katrin Kirchhoff, and George Karypis. Autogluon-multimodal (automm): Supercharging multimodal automl with foundation models. arXiv preprint arXiv:2404.16233, 2024. [117] Zhiqiang Tang, Zihan Zhong, Tong He, and Gerald Friedland. Bag of tricks for multimodal automl with image, text, and tabular data. arXiv preprint arXiv:2412.16243, 2024. [118] Myung Jun Kim, Leo Grinsztajn, and Gael Varoquaux. Carte: Pretraining and transfer for tabular learning. In International Conference on Machine Learning, pages 23843–23866. PMLR, 2024. [119] Myung Jun Kim, Félix Lefebvre, Gaëtan Brison, Alexandre Perez-Lebel, and Gaël Varoquaux. Table foundation models: on knowledge pre-training for tabular learning. arXiv preprint arXiv:2505.14415, 2025. [120] Liam Ressel and Hamza AA Gardi. Linear dimensionality reduction for word embeddings in tabular data classification. arXiv preprint arXiv:2509.12346, 2025. [121] Oksana Kolomenko, Ricardo Knauer, and Erik Rodner. Embedding world knowledge into tabular models: Towards best practices for embedding pipeline design. arXiv preprint arXiv:2603.17737, 2026. [122] Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. Tabert: Pretraining for joint understanding of textual and tabular data. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 8413–8426, 2020. [123] Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. Tabllm: Few-shot classification of tabular data with large language models. In International conference on artificial intelligence and statistics, pages 5549–5581. PMLR, 2023. [124] Xi Fang, Weijie Xu, Fiona Anting Tan, Jiani Zhang, Ziqing Hu, Yanjun Qi, Scott Nickleach, Diego Socolinsky, Srinivasan Sengamedu, and Christos Faloutsos. Large language models (llms) on tabular data: Prediction, generation, and understanding–a survey. arXiv preprint arXiv:2402.17944, 2024. [125] Canyu Chen, Jian Yu, Shan Chen, Che Liu, Zhongwei Wan, Shuang Zhou, Yuan Luo, Rui Zhang, Danielle Bitterman, Fei Wang, and Kai Shu. Clinicalbench: Can llms beat traditional ml models in clinical prediction?, 2026. URL https://arxiv.org/abs/2411.06469. [126] Sebastian Bordt, Harsha Nori, Vanessa Cristiny Rodrigues Vasconcelos, Besmira Nushi, and Rich Caruana. Elephants never forget: Memorization and learning of tabular data in large language models. In First Conference on Language Modeling, 2024. [127] Aliaksandra Shysheya, John Bronskill, James Requeima, Shoaib Ahmed Siddiqui, Javier Gonzalez, David Duvenaud, and Richard E Turner. Jolt: Joint probabilistic predictions on tabular data using llms. arXiv preprint arXiv:2502.11877, 2025. [128] Matteo Silvestri, Fabiano Veglianti, Flavio Giorgi, Fabrizio Silvestri, and Gabriele Tolomei. Evaluating latent knowledge of public tabular datasets in large language models. arXiv preprint arXiv:2510.20351, 2025. 17
[129] Hejia Liu, Mochen Yang, and Gediminas Adomavicius. Robustness is important: Limitations of llms for data fitting. arXiv preprint arXiv:2508.19563, 2025. [130] Nikolaos Pavlidis, Vasilis Perifanis, Symeon Symeonidis, and Pavlos S Efraimidis. Large language models as universal predictors? an empirical study on small tabular datasets. arXiv preprint arXiv:2508.17391, 2025. [131] Joshua P Gardner, Juan Carlos Perdomo, and Ludwig Schmidt. Large scale transfer learning for tabular data via language modeling. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. [132] Tong Lin, Jason Yan, David Jurgens, and Sabina J Tomkins. Tab2text-a framework for deep learning with tabular data. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 12925–12935, 2024. [133] Junjie Xing, Yeye He, Mengyu Zhou, Haoyu Dong, Shi Han, Dongmei Zhang, and Surajit Chaudhuri. Table-llm-specialist: Language model specialists for tables using iterative generator-validator fine-tuning. arXiv preprint arXiv:2410.12164, 2024. [134] Günther Schindler, Maximilian Schambach, Michael Medek, and Sam Thelin. Tabgemma: Text-based tabular icl via llm using continued pretraining and retrieval. arXiv preprint arXiv:2511.03570, 2025. [135] Aditya Gorla and Ratish Puduppully. The illusion of generalization in tabular language models, 2026. URL https://arxiv.org/abs/2602.04031. [136] Jiaqi Luo, Yuan Yuan, and Shixin Xu. Time: Tabpfn-integrated multimodal engine for robust tabular-image learning. arXiv preprint arXiv:2506.00813, 2025. [137] Nick Erickson, Xingjian Shi, James Sharpnack, and Alexander Smola. Multimodal automl for image, text and tabular data. In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining, pages 4786–4787, 2022. [138] Weichen Huang. Multimodal contrastive learning and tabular attention for automated alzheimer’s disease prediction. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2473–2482, 2023. [139] Siyi Du, Xinzhe Luo, Declan P O’Regan, and Chen Qin. Stil: Semi-supervised tabular-image learning for comprehensive task-relevant information exploration in multimodal classification. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 15549– 15559, 2025. [140] Paul Hager, Martin J Menten, and Daniel Rueckert. Best of both worlds: Multimodal contrastive learning with tabular and imaging data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23924–23935, 2023. [141] Tom Nuno Wolf, Sebastian Pölsterl, Christian Wachinger, Alzheimer’s Disease Neuroimaging Initiative, et al. Daft: A universal module to interweave tabular data and 3d images in cnns. NeuroImage, 260:119505, 2022. [142] Jun-Peng Jiang, Han-Jia Ye, Leye Wang, Yang Yang, Yuan Jiang, and De-Chuan Zhan. Tabular insights, visual impacts: transferring expertise from tables to images. In Forty-first International Conference on Machine Learning, 2024. [143] Sebastian Pölsterl, Tom Nuno Wolf, and Christian Wachinger. Combining 3d image and tabular data via the dynamic affine feature map transform. In International conference on medical image computing and computer-assisted intervention, pages 688–698. Springer, 2021. [144] Siyi Du, Shaoming Zheng, Yinsong Wang, Wenjia Bai, Declan P O’Regan, and Chen Qin. Tip: Tabular-image pre-training for multimodal classification with incomplete data. In European Conference on Computer Vision, pages 478–496. Springer, 2024. [145] Mafalda Malafaia, Thalea Schlender, Peter AN Bosman, and Tanja Alderliesten. Learning multimodal explainable ai models from medical images and tabular data: proof of concept. In Medical Imaging 2025: Image Processing, volume 13406, pages 263–269. SPIE, 2025. [146] Guohui Ding, Yongqiang Ren, and Zhonghua Li. Mrlf: Multimodal representation learning for tabular data classification. In 2025 International Joint Conference on Neural Networks (IJCNN), pages 1–9. IEEE, 2025. 18
[147] Marta Hasny, Laura Daza, Keno Bressem, Maxime Di Folco, and Julia Schnabel. No data? no problem: Robust vision-tabular learning with missing values. arXiv preprint arXiv:2512.19602, 2025. [148] Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The uea multivariate time series classification archive, 2018. arXiv preprint arXiv:1811.00075, 2018. [149] Chang Wei Tan, Christoph Bergmeir, François Petitjean, and Geoffrey I Webb. Time series extrinsic regression: Predicting numeric values from time series data. Data Mining and Knowledge Discovery, 35(3):1032–1060, 2021. [150] Hoang Anh Dau, Anthony Bagnall, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, and Eamonn Keogh. The ucr time series archive. IEEE/CAA Journal of Automatica Sinica, 6(6):1293–1305, 2019. [151] Matthew Middlehurst, Aiden Rushbrooke, Ali Ismail-Fawaz, Maxime Devanne, Germain Forestier, Angus Dempster, Geoffrey I Webb, Christopher Holder, and Anthony Bagnall. The multiverse of time series machine learning: an archive for multivariate time series classification. arXiv preprint arXiv:2603.20352, 2026. [152] Angus Dempster, François Petitjean, and Geoffrey I Webb. Rocket: exceptionally fast and accurate time series classification using random convolutional kernels. arXiv preprint arXiv:1910.13051, 2019. [153] Jason Lines, Sarah Taylor, and Anthony Bagnall. Time series classification with hive-cote: The hierarchical vote collective of transformation-based ensembles. ACM Transactions on Knowledge Discovery from Data (TKDD), 12(5):1–35, 2018. [154] Angus Dempster, Daniel F Schmidt, and Geoffrey I Webb. Minirocket: A very fast (almost) deterministic transform for time series classification. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pages 248–257, 2021. [155] Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski. Moment: A family of open time-series foundation models. arXiv preprint arXiv:2402.03885, 2024. [156] Vasilii Feofanov, Songkang Wen, Marius Alonso, Romain Ilbert, Hongbo Guo, Malik Tiomoko, Lujia Pan, Jianfeng Zhang, and Ievgen Redko. Mantis: Lightweight calibrated foundation model for user-friendly time series classification. arXiv preprint arXiv:2502.15637, 2025. [157] Roger M Stein. Benchmarking default prediction models: Pitfalls and remedies in model validation. Moody’s KMV, New York, 20305:16, 2002. [158] Guy Shani and Asela Gunawardana. Evaluating recommendation systems. In Recommender systems handbook, pages 257–297. Springer, 2010. [159] Christoph Bergmeir and José M Benítez. On the use of cross-validation for time series predictor evaluation. Information Sciences, 191:192–213, 2012. [160] Chip Huyen. Designing machine learning systems. " O’Reilly Media, Inc.", 2022. [161] Christina X Ji, Ahmed M Alaa, and David Sontag. Large-scale study of temporal shift in health insurance claims. In Conference on Health, Inference, and Learning, pages 243–278. PMLR, 2023. [162] Dmitry Baranchuk, Matthijs Douze, Yash Upadhyay, and I Zeki Yalniz. Dedrift: Robust similarity search under content drift. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11026–11035, 2023. [163] Francisco Ambrosio Garcia and Frank Naets. Beyond limits: enhancing the extrapolation performance of regression models by leaving the boundary out. Machine Learning, 114(12): 285, 2025. [164] Federico Garcia Crespi, Eduardo Yubero Funes, and Marina Alfosea Simon. Rolling-origin validation reverses model rankings in multi-step pm10 forecasting: Xgboost, sarima, and persistence. arXiv e-prints, pages arXiv–2603, 2026. [165] Hao-Run Cai and Han-Jia Ye. Feature-aware modulation for learning from temporal tabular data. arXiv preprint arXiv:2512.03678, 2025. 19
[166] Hao-Run Cai and Han-Jia Ye. Understanding the limits of deep tabular methods with temporal shift. arXiv preprint arXiv:2502.20260, 2025. [167] Peter Diggle. Analysis of longitudinal data. Oxford university press, 2002. [168] Robert E Weiss. Modeling longitudinal data. Springer, 2005. [169] Peter J Diggle and David Taylor-Robinson. Longitudinal data analysis. In Handbook of epidemiology, pages 1–34. Springer, 2024. [170] Geert Molenberghs and Geert Verbeke. Models for discrete longitudinal data. Springer, 2005. [171] Kevin Tierney and Yuri Malitsky. An algorithm selection benchmark of the container premarshalling problem. In International Conference on Learning and Intelligent Optimization, pages 17–22. Springer, 2015. [172] Bernd Bischl, Pascal Kerschke, Lars Kotthoff, Marius Lindauer, Yuri Malitsky, Alexandre Fréchette, Holger Hoos, Frank Hutter, Kevin Leyton-Brown, Kevin Tierney, et al. Aslib: A benchmark library for algorithm selection. Artificial Intelligence, 237:41–58, 2016. [173] Gašper Petelin and Gjorgjina Cenikj. The pitfalls of benchmarking in algorithm selection: What we are getting wrong. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 1181–1189, 2025. [174] Andrey Malinin, Neil Band, German Chesnokov, Yarin Gal, Mark JF Gales, Alexey Noskov, Andrey Ploskonosov, Liudmila Prokhorenkova, Ivan Provilkov, Vatsal Raina, et al. Shifts: A dataset of real distributional shift across multiple large-scale tasks. arXiv preprint arXiv:2107.07455, 2021. [175] Andrey Malinin, Andreas Athanasopoulos, Muhamed Barakovic, Meritxell Bach Cuadra, Mark JF Gales, Cristina Granziera, Mara Graziani, Nikolay Kartashev, Konstantinos Kyriakopoulos, Po-Jui Lu, et al. Shifts 2.0: Extending the dataset of real distributional shifts. arXiv preprint arXiv:2206.15407, 2022. [176] Jiashuo Liu, Tianyu Wang, Peng Cui, and Hongseok Namkoong. On the need for a language describing distribution shifts: Illustrations on tabular datasets. Advances in Neural Information Processing Systems, 36:51371–51408, 2023. [177] Josh Gardner, Zoran Popovic, and Ludwig Schmidt. Benchmarking distribution shift in tabular data with tableshift. Advances in Neural Information Processing Systems, 36:53385–53432, 2023. [178] Huaxiu Yao, Caroline Choi, Bochuan Cao, Yoonho Lee, Pang Wei W Koh, and Chelsea Finn. Wild-time: A benchmark of in-the-wild distribution shift over time. Advances in Neural Information Processing Systems, 35:10309–10324, 2022. [179] Sergey Kolesnikov. Wild-tab: A benchmark for out-of-distribution generalization in tabular regression. arXiv preprint arXiv:2312.01792, 2023. [180] Alan Arazi, Eilam Shapira, and Roi Reichart. Tabstar: A tabular foundation model for tabular data with text fields. Advances in Neural Information Processing Systems, 38:172108–172161, 2026. [181] N. Erickson, J. Mueller, A. Shirkov, H. Zhang, P. Larroy, M. Li, and A. Smola. Autogluontabular: Robust and accurate automl for structured data. arXiv:2003.06505 [stat.ML], 2020. [182] Arthur Asuncion, David Newman, et al. Uci machine learning repository, 2007. [183] J. Vanschoren, J. van Rijn, B. Bischl, and L. Torgo. OpenML: Networked science in machine learning. SIGKDD Explor. Newsl., 15(2):49–60, 2014. [184] Bernd Bischl, Giuseppe Casalicchio, Taniya Das, Matthias Feurer, Sebastian Fischer, Pieter Gijsbers, Subhaditya Mukherjee, Andreas C Müller, László Németh, Luis Oala, et al. Openml: Insights from 10 years and more than a thousand papers. Patterns, 2025. [185] Hugging Face, Inc. Hugging face. https://huggingface.co, 2026. [186] Kaggle, Inc. Kaggle. https://www.kaggle.com, 2026. [187] Zindi Africa. Zindi. https://zindi.africa, 2026. [188] L. Breiman. Random forests. Machine Learning, 45:5–32, 2001. 20
[189] P. Geurts, D. Ernst, and L. Wehenkel. Extremely randomized trees. Machine Learning Journal, 63(1):3–42, 2006. [190] L. Prokhorenkova, G. Gusev, A. Vorobev, A. Dorogush, and A. Gulin. Catboost: Unbiased boosting with categorical features. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Proceedings of the 31st International Conference on Advances in Neural Information Processing Systems (NeurIPS’18), page 6639–6649. Curran Associates, 2018. [191] G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.-Y. Liu. Lightgbm: A highly efficient gradient boosting decision tree. In I. Guyon, U. von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Proceedings of the 31st International Conference on Advances in Neural Information Processing Systems (NeurIPS’17). Curran Associates, 2017. [192] T. Chen and C. Guestrin. XGBoost: A scalable tree boosting system. In B. Krishnapuram, M. Shah, A. Smola, C. Aggarwal, D. Shen, and R. Rastogi, editors, Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’16), pages 785–794. ACM Press, 2016. [193] Junwei Ma, Valentin Thomas, Rasa Hosseinzadeh, Hamidreza Kamkari, Alex Labach, Jesse C Cresswell, Keyvan Golestan, Guangwei Yu, Maksims Volkovs, and Anthony L Caterini. Tabdpt: Scaling tabular foundation models. arXiv preprint arXiv:2410.18164, 2024. [194] Léo Grinsztajn, Klemens Flöge, Oscar Key, Felix Birkel, Philipp Jund, Brendan Roof, Benjamin Jäger, Dominik Safaric, Simone Alessi, Adrian Hayler, et al. Tabpfn-2.5: Advancing the state of the art in tabular foundation models. arXiv preprint arXiv:2511.08667, 2025. [195] Prior Labs. Tabpfn-2.6. https://huggingface.co/Prior-Labs/tabpfn_2_6, 2026. [196] Jingang Qu, David Holzmüller, Gaël Varoquaux, and Marine Le Morvan. Tabiclv2: A better, faster, scalable, and open tabular foundation model. arXiv preprint arXiv:2602.11139, 2026. [197] TabArena. Tabarena leaderboard for predictive machine learning on iid tabular data. https: //huggingface.co/spaces/TabArena/leaderboard, 2025. [198] T. Nagler, L. Schneider, B. Bischl, and M. Feurer. Reshuffling resampling splits can improve generalization of hyperparameter optimization. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Proceedings of the 37th International Conference on Advances in Neural Information Processing Systems (NeurIPS’24). Curran Associates, 2024. [199] Lennart Schneider, Bernd Bischl, and Matthias Feurer. Overtuning in hyperparameter optimization. In International Conference on Automated Machine Learning, 2025. [200] Jaris Küken, Lennart Purucker, and Frank Hutter. Early stopping tabular in-context learning. In 1st ICML Workshop on Foundation Models for Structured Data, 2025. [201] skrub developers. skrub: Machine learning with dataframes, 2026. URL https://github. com/skrub-data/skrub. [202] Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176, 2025. [203] Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. Mteb: Massive text embedding benchmark. arXiv preprint arXiv:2210.07316, 2022. doi: 10.48550/ARXIV.2210. 07316. [204] Kenneth Enevoldsen, Isaac Chung, Imene Kerboua, Márton Kardos, Ashwin Mathur, David Stap, Jay Gala, Wissam Siblini, Dominik Krzemiński, Genta Indra Winata, Saba Sturua, Saiteja Utpala, Mathieu Ciancone, Marion Schaeffer, Gabriel Sequeira, Diganta Misra, Shreeya Dhakal, Jonathan Rystrøm, Roman Solomatin, Ömer Çağatan, Akash Kundu, Martin Bernstorff, Shitao Xiao, Akshita Sukhlecha, Bhavish Pahwa, Rafał Poświata, Kranthi Kiran GV, Shawon Ashraf, Daniel Auras, Björn Plüster, Jan Philipp Harries, Loïc Magne, Isabelle Mohr, Mariya Hendriksen, Dawei Zhu, Hippolyte Gisserot-Boukhlef, Tom Aarsen, Jan Kostkan, Konrad Wojtasik, Taemin Lee, Marek Šuppa, Crystina Zhang, Roberta Rocca, Mohammed Hamdy, Andrianos Michail, John Yang, Manuel Faysse, Aleksei Vatolin, Nandan Thakur, Manan 21
Dey, Dipam Vasani, Pranjal Chitale, Simone Tedeschi, Nguyen Tai, Artem Snegirev, Michael Günther, Mengzhou Xia, Weijia Shi, Xing Han Lù, Jordan Clive, Gayatri Krishnakumar, Anna Maksimova, Silvan Wehrli, Maria Tikhonova, Henil Panchal, Aleksandr Abramov, Malte Ostendorff, Zheng Liu, Simon Clematide, Lester James Miranda, Alena Fenogenova, Guangyu Song, Ruqiya Bin Safi, Wen-Ding Li, Alessia Borghini, Federico Cassano, Hongjin Su, Jimmy Lin, Howard Yen, Lasse Hansen, Sara Hooker, Chenghao Xiao, Vaibhav Adlakha, Orion Weller, Siva Reddy, and Niklas Muennighoff. Mmteb: Massive multilingual text embedding benchmark. arXiv preprint arXiv:2502.13595, 2025. doi: 10.48550/arXiv.2502.13595. [205] Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, et al. Matryoshka representation learning. Advances in Neural Information Processing Systems, 35: 30233–30249, 2022. [206] Addison Howard, AritraAmex, Di Xu, Hossein Vashani, inversion, Negin, and Sohier Dane. American express - default prediction. https://kaggle.com/competitions/ amex-default-prediction, 2022. Kaggle competition. [207] Google LLC. Google cloud platform, 2026. URL https://cloud.google.com/. [208] Foster J Provost, Tom Fawcett, Ron Kohavi, et al. The case against accuracy estimation for comparing induction algorithms. In ICML, volume 98, pages 445–453, 1998. [209] Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477):359–378, 2007. [210] Tobias Fissler, Christian Lorentzen, and Michael Mayer. Model comparison and calibration assessment: User guide for consistent scoring functions in machine learning and actuarial practice. arXiv preprint arXiv:2202.12780, 2022. [211] Tilmann Gneiting. Making and evaluating point forecasts. Journal of the American Statistical Association, 106(494):746–762, 2011. [212] Tilmann Gneiting and Matthias Katzfuss. Probabilistic forecasting. Annual Review of Statistics and Its Application, 1(1):125–151, 2014. [213] Jonas Landsgesell, Pascal Knoll, and Tizian Wenzel. Scoringbench: A benchmark for evaluating tabular foundation models with proper scoring rules. arXiv preprint arXiv:2603.29928, 2026. [214] Arpad E Elo. The proposed uscf rating system, its development, theory, and applications. Chess life, 22(8):242–247, 1967. [215] Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1-2):81–93, 1938. [216] A. Cimatti and R. Sebastiani, editors. Proceedings of the Fifteenth International Conference on Theory and Applications of Satisfiability Testing (SAT’12), volume 7317 of Lecture Notes in Computer Science, 2012. Springer. [217] agentskills. agentskills, 2026. URL https://github.com/agentskills/agentskills. GitHub repository. [218] Anthropic. Claude code, 2026. URL https://github.com/anthropics/claude-code. GitHub repository. [219] Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. Deepfm: A factorization-machine based neural network for ctr prediction. In 26th International Joint Conference on Artificial Intelligence, IJCAI 2017, pages 1725–1731. International Joint Conferences on Artificial Intelligence, 2017. [220] Jieming Zhu, Jinyang Liu, Shuai Yang, Qi Zhang, and Xiuqiang He. Open benchmarking for click-through rate prediction. In Proceedings of the 30th ACM international conference on information & knowledge management, pages 2759–2769, 2021. [221] Weinan Zhang, Jiarui Qin, Wei Guo, Ruiming Tang, and Xiuqiang He. Deep learning for click-through rate estimation. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, pages 4695–4703. International Joint Conferences on Artificial Intelligence Organization, 2021. [222] Harald Steck. Evaluation of recommendations: rating-prediction and ranking. In Proceedings of the 7th ACM conference on Recommender systems, pages 213–220, 2013. 22
[223] Alan Said and Alejandro Bellogín. Comparative recommender system evaluation: benchmarking recommendation frameworks. In Proceedings of the 8th ACM Conference on Recommender systems, pages 129–136, 2014. [224] Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay. Deep learning based recommender system: A survey and new perspectives. ACM computing surveys (CSUR), 52(1):1–38, 2019. [225] Pierre Baldi, Peter Sadowski, and Daniel Whiteson. Searching for exotic particles in highenergy physics with deep learning. Nature communications, 5(1):4308, 2014. [226] Claire Adam-Bourdarios, Glen Cowan, Cécile Germain, Isabelle Guyon, Balázs Kégl, and David Rousseau. The higgs boson machine learning challenge. In NIPS 2014 workshop on high-energy physics and machine learning, pages 19–55. PMLR, 2015. [227] Wahid Bhimji, Ragansu Chakkappai, Po-Wen Chang, Yuan-Tang Chou, Sascha Diefenbacher, Jordan Dudley, Ibrahim Elsharkawy, Steven Farrell, Aishik Ghosh, Cristina Giordano, et al. Fair universe higgsml uncertainty dataset and competition. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025. [228] Joanna Radin. “digital natives”: how medical and indigenous histories matter for big data. Osiris, 32(1):43–64, 2017. [229] Bradley Efron and Gail Gong. Statistical theory and the computer. In Computer science and statistics: Proceedings of the 13th Symposium on the Interface, pages 3–7. Springer, 1981. [230] E Rolland Dickson, Patricia M Grambsch, Thomas R Fleming, Lloyd D Fisher, and Alice Langworthy. Prognosis in primary biliary cirrhosis: model for decision making. Hepatology, 10(1):1–7, 1989. [231] Seref Gul, Fatih Rahim, Safak Isin, Fatma Yilmaz, Nuri Ozturk, Metin Turkay, and Ibrahim Halil Kavakli. Structure-based design and classifications of small molecules regulating the circadian rhythm period. Scientific reports, 11(1):18510, 2021. [232] Sunil R Hingorani, Emanuel F Petricoin, Anirban Maitra, Vinodh Rajapakse, Catrina King, Michael A Jacobetz, Sally Ross, Thomas P Conrads, Timothy D Veenstra, Ben A Hitt, et al. Preinvasive and invasive ductal pancreatic cancer and its early detection in the mouse. Cancer cell, 4(6):437–450, 2003. [233] Avrum Spira, Jennifer E Beane, Vishal Shah, Katrina Steiling, Gang Liu, Frank Schembri, Sean Gilman, Yves-Martine Dumas, Paul Calner, Paola Sebastiani, et al. Airway epithelial gene expression in the diagnostic evaluation of smokers with suspect lung cancer. Nature medicine, 13(3):361–366, 2007. [234] Max Little, Patrick Mcsharry, Stephen Roberts, Declan Costello, and Irene Moroz. Exploiting nonlinear recurrence and fractal scaling properties for voice disorder detection. Nature Precedings, pages 1–1, 2007. [235] Arindam Bhattacharjee, William G Richards, Jane Staunton, Cheng Li, Stefano Monti, Priya Vasa, Christine Ladd, Javad Beheshti, Raphael Bueno, Michael Gillette, et al. Classification of human lung carcinomas by mrna expression profiling reveals distinct adenocarcinoma subclasses. Proceedings of the National Academy of Sciences, 98(24):13790–13795, 2001. [236] E Ray Bareiss, Bruce W Porter, and Craig C Wier. Protos: An exemplar-based learning apprentice. In Machine learning, pages 112–127. Elsevier, 1990. [237] Robert Detrano, Andras Janosi, Walter Steinbrunn, Matthias Pfisterer, Johann-Jakob Schmid, Sarbjit Sandhu, Kern H Guppy, Stella Lee, and Victor Froelicher. International application of a new probability algorithm for the diagnosis of coronary artery disease. The American journal of cardiology, 64(5):304–310, 1989. [238] B. German. Glass Identification. UCI Machine Learning Repository, 1987. DOI: https://doi.org/10.24432/C5WW2P. [239] MM Faniqul Islam, Rahatara Ferdousi, Sadikur Rahman, and Humayra Yasmin Bushra. Likelihood prediction of diabetes at early stage using data mining techniques. In Computer Vision and Machine Intelligence in Medical Image Analysis: International Symposium, ISCMM 2019, pages 113–125. Springer, 2019. [240] Keith W Penrose, Arnold G Nelson, and Arnold Garth Fisher. Generalized body composition prediction equation for men using simple measurement techniques. Medicine & Science in Sports & Exercise, 17(2):189, 1985. 23
[241] Matjaz Zwitter and Milan Soklic. Breast Cancer. UCI Machine Learning Repository, 1988. DOI: https://doi.org/10.24432/C51P4M. [242] Davide Chicco and Giuseppe Jurman. Machine learning can predict survival of patients with heart failure from serum creatinine and ejection fraction alone. BMC medical informatics and decision making, 20(1):16, 2020. [243] M. Zwitter and M. Soklic. Primary Tumor. UCI Machine Learning Repository, 1987. DOI: https://doi.org/10.24432/C5WK5Q. [244] Guilherme Barreto and Ajalmar Neto. Vertebral Column. UCI Machine Learning Repository, 2005. DOI: https://doi.org/10.24432/C5K89B. [245] İrfan Esen, Hilal Arslan, Selin Aktürk Esen, Mervenur Gülşen, Nimet Kültekin, and Oğuzhan Özdemir. Early prediction of gallstone disease with a machine learning-based method from bioimpedance and laboratory data. Medicine, 103(8):e37258, 2024. [246] Emanuel F Petricoin III, David K Ornstein, Cloud P Paweletz, Ali Ardekani, Paul S Hackett, Ben A Hitt, Alfredo Velassco, Christian Trucco, Laura Wiegand, Kamillah Wood, et al. Serum proteomic patterns for detection of prostate cancer. Journal of the National Cancer Institute, 94(20):1576–1578, 2002. [247] Paul Horton and Kenta Nakai. A probabilistic classification system for predicting the cellular localization sites of proteins. In Ismb, volume 4, pages 109–115. St. Louis, Missouri, USA, 1996. [248] Mary McLeish and Matt Cecile. Horse Colic. UCI Machine Learning Repository, 1989. DOI: https://doi.org/10.24432/C58W23. [249] UCI Machine Learning Repository. Liver disorders. https://doi.org/10.24432/C54G67, 2016. Dataset. [250] H Altay Güvenir, Gülşen Demiröz, and Nilsel Ilter. Learning differential diagnosis of erythemato-squamous diseases using voting feature intervals. Artificial intelligence in medicine, 13(3):147–165, 1998. [251] Daniel S Marcus, Anthony F Fotenos, John G Csernansky, John C Morris, and Randy L Buckner. Open access series of imaging studies: longitudinal mri data in nondemented and demented older adults. Journal of cognitive neuroscience, 22(12):2677–2684, 2010. [252] JE Rossouw, JP Du Plessis, AJ Benadé, PC Jordaan, JP Kotze, PL Jooste, and JJ Ferreira. Coronary risk factor screening in three rural communities. the coris baseline study. South African medical journal= Suid-Afrikaanse tydskrif vir geneeskunde, 64(12):430–436, 1983. [253] Fabio Mendoza Palechor and Alexis De la Hoz Manotas. Dataset for estimation of obesity levels based on eating habits and physical condition in individuals from colombia, peru and mexico. Data in brief, 25:104344, 2019. [254] Athanasios Tsanas, Max Little, Patrick McSharry, and Lorraine Ramig. Accurate telemonitoring of parkinson’s disease progression by non-invasive speech tests. Nature Precedings, pages 1–1, 2009. [255] Paulo Cortez and Aníbal de Jesus Raimundo Morais. A data mining approach to predict forest fires using meteorological data, 2007. [256] Matteo Cassotti, Davide Ballabio, Viviana Consonni, Andrea Mauri, Igor V Tetko, and Roberto Todeschini. Prediction of acute aquatic toxicity toward daphnia magna by using the ga-k nn method. Alternatives to Laboratory Animals, 42(1):31–41, 2014. [257] Pierre Mahe, Maud Arsac, Sonia Chatellier, Valérie Monnin, Nadine Perrot, Sandrine Mailler, Victoria Girard, Mahendrasingh Ramjeet, Jérémy Surre, Bruno Lacroix, et al. Automatic identification of mixed bacterial species fingerprints in a maldi-tof mass-spectrum. Bioinformatics, 30(9):1280–1286, 2014. [258] Bendi Venkata Ramana, M Surendra Prasad Babu, and NB Venkateswarlu. A critical comparative study of liver patients from usa and india: an exploratory analysis. International Journal of Computer Science Issues (IJCSI), 9(3):506, 2012. [259] Lina Huang, Peineng Liu, and Xiaojie Huang. Interdia: Interpretable prediction of druginduced autoimmunity through ensemble machine learning approaches. Toxicology, 511: 154064, 2025. 24
[260] Georg Hoffmann, Andreas Bietenbeck, Ralf Lichtinghagen, and Frank Klawonn. Using machine learning techniques to generate laboratory diagnostic pathways—a case study. Journal of Laboratory and Precision Medicine, 3(6), 2018. [261] Jorge Ruiz-Ramírez, M de La Puente, Catarina Xavier, Adrián Ambroa-Conde, J Álvarez-Dios, A Freire-Aradas, Ana Mosquera-Miguel, Arwin Ralf, Christina Amory, Maria Alexandra Katsara, et al. Development and evaluations of the ancestry informative markers of the visage enhanced tool for appearance and ancestry. Forensic Science International: Genetics, 64: 102853, 2023. [262] Catarina Xavier, Maria de la Puente, Ana Mosquera-Miguel, Ana Freire-Aradas, Vivian Kalamara, Athina Vidaki, Theresa E Gross, Andrew Revoir, Ewelina Pośpiech, Ewa Kartasińska, et al. Development and validation of the visage ampliseq basic tool to predict appearance and ancestry from dna. Forensic Science International: Genetics, 48:102336, 2020. [263] Alice Silva. Using data mining to predict secondary school student performance, 2008. [264] J. Ross Quinlan. Simplifying decision trees. International journal of man-machine studies, 27 (3):221–234, 1987. [265] I-Cheng Yeh, King-Jang Yang, and Tao-Ming Ting. Knowledge discovery on rfm model using bernoulli sequence. Expert Systems with applications, 36(3):5866–5871, 2009. [266] Ričards Marcinkevičs, Patricia Reis Wolfertstetter, Ugne Klimiene, Kieran Chin-Cheong, Alyssia Paschke, Julia Zerres, Markus Denzinger, David Niederberger, Sven Wellmann, Ece Ozkan, et al. Interpretable and intervenable ultrasonography-based machine learning models for pediatric appendicitis. Medical image analysis, 91:103042, 2024. [267] Ravi Barnawal. Mutual funds india detailed. https://www.kaggle.com/datasets/ ravibarnawal/mutual-funds-india-detailed, 2022. Kaggle dataset. [268] Matteo Cassotti, Davide Ballabio, Roberto Todeschini, and Viviana Consonni. A similaritybased qsar model for predicting acute toxicity towards the fathead minnow (pimephales promelas). SAR and QSAR in Environmental Research, 26(3):217–243, 2015. [269] Tejashvi. Tour & travels customer churn prediction. https://www.kaggle.com/datasets/ tejashvi14/tour-travels-customer-churn-prediction, 2023. Kaggle dataset. [270] H. Hofmann. Statlog (german credit data) [dataset]. https://doi.org/10.24432/C5NC77, 1994. UCI Machine Learning Repository. [271] Marzia Ahmed, Mohammod Abul Kashem, Mostafijur Rahman, and Sabira Khatun. Review and analysis of risk factor of maternal health in remote area using the internet of things (iot). In InECCE2019: Proceedings of the 5th International Conference on Electrical, Control & Computer Engineering, Kuantan, Pahang, Malaysia, 29th July 2019, pages 357–365. Springer, 2020. [272] I-C Yeh. Modeling of strength of high-performance concrete using artificial neural networks. Cement and Concrete research, 28(12):1797–1808, 1998. [273] Kamel Mansouri, Tine Ringsted, Davide Ballabio, Roberto Todeschini, and Viviana Consonni. Quantitative structure–activity relationship models for ready biodegradability of chemicals. Journal of chemical information and modeling, 53(4):867–878, 2013. [274] Clara Higuera, Katheleen J Gardiner, and Krzysztof J Cios. Self-organizing feature maps identify proteins critical to learning in a mouse model of down syndrome. PloS one, 10(6): e0129126, 2015. [275] Abdullah Al Imran, Md Shamsur Rahim, and Tanvir Ahmed. Mining the productivity data of the garment industry. International Journal of Business Intelligence and Data Mining, 19(3): 319–342, 2021. [276] Holger Hoos, Marius Lindauer, and Torsten Schaub. claspfolio 2: Advances in algorithm selection for answer set programming. Theory and Practice of Logic Programming, 14(4-5): 569–585, 2014. [277] Bernd Bischl, Pascal Kerschke, Lars Kotthoff, Marius Lindauer, Yuri Malitsky, Alexandre Fréchette, Holger Hoos, Frank Hutter, Kevin Leyton-Brown, Kevin Tierney, et al. Aslib: A benchmark library for algorithm selection. Artificial Intelligence, 237:41–58, 2016. 25
[278] Elvin Rustamov. Wine dataset. https://www.kaggle.com/datasets/elvinrustam/ wine-dataset, 2023. Kaggle dataset. [279] Kaggle User Arunjangir245. Healthcare insurance expenses. https://www.kaggle.com/ datasets/arunjangir245/healthcare-insurance-expenses/, 2023. Kaggle dataset. [280] Neda Abdelhamid, Aladdin Ayesh, and Fadi Thabtah. Phishing detection based associative classification data mining. Expert Systems with Applications, 41(13):5948–5959, 2014. [281] Kaggle User Ddosad. Fitness club dataset for ml classification. https://www.kaggle.com/ 2023. datasets/ddosad/datacamps-data-science-associate-certification, Kaggle dataset. [282] Thomas F Brooks, D Stuart Pope, and Michael A Marcolini. Airfoil self-noise and prediction, 1989. [283] Kaggle User Paolocons. Another dataset on used fiat 500 (1538 rows). https://www.kaggle. com/datasets/paolocons/another-fiat-500-dataset-1538-rows, 2020. Kaggle dataset. [284] Sergey E Golovenkin, Jonathan Bac, Alexander Chervov, Evgeny M Mirkes, Yuliya V Orlova, Emmanuel Barillot, Alexander N Gorban, and Andrei Zinovyev. Trajectories, bifurcations, and pseudo-time in large clinical datasets: applications to myocardial infarction and diabetes data. GigaScience, 9(11):giaa128, 2020. [285] Podsyp. Is this a good customer?, 2020. URL https://www.kaggle.com/datasets/ podsyp/is-this-a-good-customer. [286] D. Campos and J. Bernardes. Cardiotocography. UCI Machine Learning Repository, 2000. DOI: https://doi.org/10.24432/C51S4N. Marketing campaign. https://www.kaggle.com/datasets/ [287] Rodolfo Saldanha. rodsaldanha/arketing-campaign, 2020. Kaggle dataset. [288] Hanif Al Irsyad. Coffee data coffeereview. https://www.kaggle.com/datasets/ hanifalirsyad/coffee-scrap-coffeereview, 2023. Kaggle dataset. [289] Marco Ricci, Bernardita Štitić, Luca Urbinati, Giuseppe Di Guglielmo, Jorge A Tobón Vasquez, Luca P Carloni, Francesca Vipiana, and Mario R Casu. Machine-learning-based microwave sensing: A case study for the food industry. IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 11(3):503–514, 2021. [290] Marek Sikora and Łukasz Wróbel. Application of rule induction algorithms for analysis of data collected by seismic hazard monitoring systems in coal mines. Archives of Mining Sciences, 55(1):91–114, 2010. [291] Abbas Keramati and Seyed MS Ardabili. Churn analysis for an iranian mobile operator. Telecommunications Policy, 35(4):344–356, 2011. [292] L. Xu, F. Hutter, H. Hoos, and K. Leyton-Brown. Evaluating component solver contributions to portfolio-based algorithm selectors. In Cimatti and Sebastiani [216], pages 228–241. [293] Geoffrey G Towell and Jude W Shavlik. Knowledge-based artificial neural networks. Artificial intelligence, 70(1-2):119–165, 1994. [294] Ben Hamner, dcthompson, and Jorg. Predicting https://kaggle.com/competitions/bioresponse, 2012. Kaggle.
a
biological
response.
[295] Isabelle Guyon, Amir Saffari, Gideon Dror, and Gavin Cawley. Agnostic learning vs. prior knowledge challenge. In 2007 International Joint Conference on Neural Networks, pages 829–834, 2007. doi: 10.1109/IJCNN.2007.4371065. [296] Alexander Novy, CH1Mercedes, Christian Drescher, Christian Pfaundler, KOESIM, and Will Cukierski. Mercedes-benz greener manufacturing. https://kaggle.com/competitions/ mercedes-benz-greener-manufacturing, 2017. Kaggle competition. [297] Mónica V Martins, Daniel Tolledo, Jorge Machado, Luís MT Baptista, and Valentim Realinho. Early prediction of student’s performance in higher education: A case study. In Trends and Applications in Information Systems and Technologies: Volume 1 9, pages 166–175. Springer, 2021. 26
[298] Mark McDonald, Mercedes Piedra, Sohier Dane, and Soraya Jimenez. Santander value prediction challenge. https://kaggle.com/competitions/ santander-value-prediction-challenge, 2018. Kaggle competition. [299] George A Marcoulides. Discovering knowledge in data: An introduction to data mining, 2005. [300] Bart Baesens, Daniel Roesch, and Harald Scheule. Credit risk analytics: Measurement techniques, applications, and examples in SAS. John Wiley & Sons, 2016. [301] Ivan Olier, Noureddin Sadawi, G Richard Bickerton, Joaquin Vanschoren, Crina Grosan, Larisa Soldatova, and Ross D King. Meta-qsar: a large-scale application of meta-learning to drug design and discovery. Machine Learning, 107:285–311, 2018. [302] Maciej Zi˛eba, Sebastian K Tomczak, and Jakub M Tomczak. Ensemble boosted trees with synthetic features generation in application to bankruptcy prediction. Expert systems with applications, 58:93–101, 2016. [303] Paulo Cortez, António Cerdeira, Fernando Almeida, Telmo Matos, and José Reis. Modeling wine preferences by data mining from physicochemical properties. Decision support systems, 47(4):547–553, 2009. [304] Thomas Dietterich, Ajay Jain, Richard Lathrop, and Tomas Lozano-Perez. A comparison of dynamic reposing and tangent distance for drug activity prediction. Advances in neural information processing systems, 6, 1993. [305] Deron Liang, Chia-Chi Lu, Chih-Fong Tsai, and Guan-An Shih. Financial ratios and corporate governance indicators in bankruptcy prediction: A comprehensive study. European journal of operational research, 252(2):561–572, 2016. [306] Akshay Mathur, Laxmi Mounika Podila, Keyur Kulkarni, Quamar Niyaz, and Ahmad Y Javaid. Naticusdroid: A malware detection framework for android using native and custom permissions. Journal of Information Security and Applications, 58:102696, 2021. [307] Peter Van Der Putten, Maarten van Someren, et al. Coil challenge 2000: The insurance company case. Technical report, Technical Report 2000–09, Leiden Institute of Advanced Computer Science, 2000. [308] Gaurav Topre. Bank customer churn dataset, 2022. URL https://www.kaggle.com/ datasets/gauravtopre/bank-customer-churn-dataset. [309] shritech1404. German housing price prediction. https://www.kaggle.com/code/ shritech1404/german-housing-price-prediction, 2019. Kaggle notebook. [310] OpenML. Immoscout24 openml dataset 43342. https://www.openml.org/d/43342, 2023. OpenML dataset. [311] Kaggle User Averkiyoliabev. Home equity line of credit (heloc). https://www.kaggle. com/datasets/averkiyoliabev/home-equity-line-of-creditheloc, 2021. Kaggle dataset. [312] Tim Menzies and Justin S Di Stefano. How good is your blind spot sampling policy. In Eighth IEEE International Symposium on High Assurance Systems Engineering, 2004. Proceedings., pages 129–138. IEEE, 2004. [313] Zindi. Ghana’s indigenous intel challenge [beginners only]: Data. https://zindi.africa/ competitions/ghana-indigenous-intel-challenge/data, 2025. Zindi dataset page. Accessed 2026-04-11. [314] Prachi Gopalani. E-commerce shipping data. https://www.kaggle.com/datasets/ prachi13/customer-analytics, 2021. Kaggle dataset. [315] Sven Peeters, Vitalik Melnikov, and Eyke H""ullermeier. Performance prediction for hardwaresoftware configurations: A case study for video games. In International Symposium on Intelligent Data Analysis, pages 222–234. Springer, 2021. [316] C Okan Sakar, S Olcay Polat, Mete Katircioglu, and Yomi Kastro. Real-time prediction of online shoppers’ purchasing intention using multilayer perceptron and lstm recurrent neural networks. Neural Computing and Applications, 31(10):6893–6908, 2019. [317] Tong Wang, Cynthia Rudin, Finale Doshi-Velez, Yimin Liu, Erica Klampfl, and Perry MacNeille. A bayesian framework for learning rule sets for interpretable classification. Journal of Machine Learning Research, 18(70):1–37, 2017. 27
[318] Steven C Bourassa, Martin Hoesli, Louis Merlin, and John Renne. Big data, accessibility and urban house prices. Urban Studies, 58(15):3176–3195, 2021. [319] Sokratis Vidros, Constantinos Kolias, Georgios Kambourakis, and Leman Akoglu. Automatic detection of online recruitment frauds: Characteristics, methods, and a public dataset. Future Internet, 9(1):6, 2017. [320] DataDrive2030. Elom and thrive by five index 2016–2023, merged data, 2024. URL https://doi.org/10.25828/WG0D-Y909. [dataset]. Producer: DataDrive2030; Distributor: DataFirst. [321] Kaggle User Arashnic. Hr analytics: Job change of data scientists. https://www.kaggle. com/datasets/arashnic/hr-analytics-job-change-of-data-scientists, 2021. Kaggle dataset. [322] R Kelley Pace and Ronald Barry. Sparse spatial autoregressions. Statistics & Probability Letters, 33(3):291–297, 1997. [323] Kam Hamidieh. A data-driven statistical model for predicting the critical temperature of a superconductor. Computational Materials Science, 154:346–354, 2018. [324] Daniel Herman, Tomas Jelinek, Walter Reade, Maggie Demkin, and Addison Howard. Home credit - credit risk model stability. https://kaggle.com/competitions/ home-credit-credit-risk-model-stability, 2024. Kaggle competition. [325] I-Cheng Yeh and Che-hui Lien. The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients. Expert systems with applications, 36 (2):2473–2480, 2009. [326] Ben Hamner, kenmonta, and Will Cukierski. Amazon.com - employee access challenge. https://www.kaggle.com/competitions/amazon-employee-access-challenge, 2013. Kaggle competition. [327] d2lcourse. California house prices. california-house-prices, 2021. Kaggle.
https://kaggle.com/competitions/
[328] Sérgio Moro, Paulo Cortez, and Paulo Rita. A data-driven approach to predict the success of bank telemarketing. Decision Support Systems, 62:22–31, 2014. [329] Kaggle User Rajatkumar30. Food delivery time. https://www.kaggle.com/datasets/ rajatkumar30/food-delivery-time, 2023. Kaggle dataset. [330] Prashant Rana. Physicochemical properties of protein tertiary structure. https://doi.org/ 10.24432/C5QW3H, 2013. UCI Machine Learning Repository. [331] American National Election Studies. ANES Time Series Cumulative Data File [dataset and documentation]. https://www.electionstudies.org, February 2026. February 5, 2026 version. [332] Isabelle Guyon, Vincent Lemaire, Marc Boullé, Gideon Dror, and David Vogel. Analysis of the kdd cup 2009: Fast scoring on a large orange customer database. In KDD-Cup 2009 Competition, pages 1–22. PMLR, 2009. [333] Hadley Wickham. Data analysis. In ggplot2: Elegant graphics for data analysis, pages 189–201. Springer, 2016. [334] Benjamin Bossan, Josef Feigl, and Wendy Kan. Otto group product classification challenge. https://kaggle.com/competitions/ otto-group-product-classification-challenge, 2015. Kaggle competition. [335] Eirik Lund Flogard and Ole Jakob Mengshoel. A dataset for efforts towards achieving the sustainable development goal of safe working environments. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. [336] Tewodors Deneke, Habtegebreil Haile, S’ebastien Lafond, and Johan Lilius. Video transcoding time prediction for proactive load balancing. In 2014 IEEE International Conference on Multimedia and Expo (ICME), pages 1–6. IEEE, 2014. [337] Soraya Jimenez and Will Cukierski. Santander customer satisfaction. https://kaggle.com/ competitions/santander-customer-satisfaction, 2016. Kaggle competition. 28
[338] Beata Strack, Jonathan P DeShazo, Chris Gennings, Juan L Olmo, Sebastian Ventura, Krzysztof J Cios, and John N Clore. Impact of hba1c measurement on hospital readmission rates: analysis of 70,000 clinical database patient records. BioMed research international, 2014(1):781670, 2014. [339] faysal, Will Adams, and Will Cukierski. Don’t get kicked! competitions/DontGetKicked, 2011. Kaggle.
https://kaggle.com/
[340] IDA2016Challenge. Ida2016challenge [dataset]. https://doi.org/10.24432/C5V60Q, 2016. UCI Machine Learning Repository. [341] Katherine Accetta, Conny Aerts, Victor Silva Aguirre, Romina Ahumada, Nikhil Ajgaonkar, N Filiz Ak, Shadab Alam, Carlos Allende Prieto, Andres Almeida, Friedrich Anders, et al. The seventeenth data release of the sloan digital sky surveys: Complete release of manga, mastar, and apogee-2 data. The Astrophysical Journal Supplement Series, 259(2):35, 2022. [342] Nuno Antonio, Ana de Almeida, and Luis Nunes. Hotel booking demand datasets. Data in brief, 22:41–49, 2019. [343] HUAWEI Netop Team. 5g network energy consumption dataset. https://huggingface.co/ datasets/netop/5G-Network-Energy-Consumption, n.d. Dataset hosted on Hugging Face, accessed 2026-04-15. [344] Davide Chicco and Giuseppe Jurman. Survival prediction of patients with sepsis from age, sex, and septic episode number alone. Scientific reports, 10(1):17156, 2020. [345] City and County of San Francisco. Building permits. https://data.sfgov.org/ Housing-and-Buildings/Building-Permits/i98e-djp9/about_data, 2026. DataSF Open Data Portal dataset, Accessed: 2026-02-05. [346] Karen Matthys, Meredith Lee, Neha Goel, Sharada Kalanidhi, Valerie, and Vani M. Wids datathon 2021. https://kaggle.com/competitions/widsdatathon2021, 2021. Kaggle competition. User Yakhyojon. Customer satisfaction in [347] Kaggle urlhttps://www.kaggle.com/datasets/yakhyojon/customer-satisfaction-in-airline, Kaggle.
airline. 2023.
[348] Ismail Parsa. Kdd cup 1998. https://kdd.ics.uci.edu/databases/kddcup98/ kddcup98.html, 1998. Dataset, UCI Machine Learning Repository. DOI: 10.24432/C5401H. [349] Credit Fusion and Will Cukierski. Give me some https://kaggle.com/competitions/GiveMeSomeCredit, 2011. Kaggle.
credit.
url-
[350] DMDave, Todd B, and Will Cukierski. Acquire valued shoppers challenge. https: //kaggle.com/competitions/acquire-valued-shoppers-challenge, 2014. Kaggle competition. [351] Web Robots. Kickstarter datasets. https://webrobots.io/kickstarter-datasets/, 2026. Accessed: 2026-01-25. [352] Dana Ferguson, Meg Risdal, NoTrick, Sara R. Sillah, Tim Emmerling, and Will Cukierski. Allstate claims severity. https://kaggle.com/competitions/ allstate-claims-severity, 2016. Kaggle competition. [353] Mercedes Piedra, Sohier Dane, and Soraya Jimenez. Santander customer transaction prediction. https://kaggle.com/competitions/ santander-customer-transaction-prediction, 2019. Kaggle competition. [354] Darrel, Stephen D. Stayton, and Will Cukierski. Homesite quote conversion. https:// kaggle.com/competitions/homesite-quote-conversion, 2015. Kaggle competition. [355] Anna Montoya, inversion, Kirill Odintsov, and Martin Kotek. Home credit default risk. https: //kaggle.com/competitions/home-credit-default-risk, 2018. Kaggle competition. [356] Jock A Blackard and Denis J Dean. Comparative accuracies of artificial neural networks and discriminant analysis in predicting forest cover types from cartographic variables. Computers and electronics in agriculture, 24(3):131–151, 1999. 29
[357] Addison Howard, Bernadette Bouchon-Meunier, IEEE CIS, inversion, John Lei, Lynn@Vesta, Marcus2010, and Prof. Hussein Abbass. Ieee-cis fraud detection. https://kaggle.com/ competitions/ieee-fraud-detection, 2019. Kaggle. [358] Addison Howard, Adriano Moala, and Walter Reade. Porto seguro’s safe driver prediction. https://kaggle.com/competitions/porto-seguro-safe-driver-prediction, 2017. Kaggle competition. [359] Kaggle. Rossmann store sales. https://www.kaggle.com/competitions/ rossmann-store-sales/overview, 2015. Kaggle competition page. Accessed: 202603-19. [360] Mario Sanz-Guerrero and Javier Arroyo. Credit risk meets large language models: Building a risk indicator from loan descriptions in p2p lending. Inteligencia Artificial, 28(75):220–247, 2025. [361] Consumer Financial Protection Bureau. Consumer complaint database. https://www. consumerfinance.gov/data-research/consumer-complaints/, 2025. Accessed: 2026-01-23. [362] Matthew A Reyna, Christopher S Josef, Russell Jeter, Supreeth P Shashikumar, M Brandon Westover, Shamim Nemati, Gari D Clifford, and Ashish Sharma. Early prediction of sepsis from clinical data: the physionet/computing in cardiology challenge 2019. Critical care medicine, 48(2):210–217, 2020. [363] Addison Howard, AritraAmex, Di Xu, Hossein Vashani, inversion, Negin, and Sohier Dane. American express – default prediction. Kaggle Competition, 2022. URL https://kaggle. com/competitions/amex-default-prediction. Accessed via Kaggle. [364] Ivan Rubachev, Nikolay Kartashev, Yury Gorishniy, and Artem Babenko. Tabred: Analyzing pitfalls and filling the gaps in tabular deep learning benchmarks. In The Thirteenth International Conference on Learning Representations, 2025. [365] Kaggle, Addison Howard, kaoriiida, Kei Otagaki, Mark McDonald, mueno, Wendy Kan, Zhang, and zyaga. Mercari price suggestion challenge. https://kaggle.com/ competitions/mercari-price-suggestion-challenge, 2017. Kaggle competition. [366] Wilhelm Kirchg"assner, Oliver Wallscheid, and Joachim B"ocker. Estimating electric motor temperatures with deep residual machine learning. IEEE Transactions on Power Electronics, 36(7):7480–7488, 2020. Accelerator-optimized machines: A2 high-gpu (a2-highgpu[367] Google LLC. 1g), 2026. URL https://docs.cloud.google.com/compute/docs/ accelerator-optimized-machines#a2-standard-vms. Accelerator-optimized machines: A2 ultra gpu (a2-ultragpu[368] Google LLC. 1g), 2026. URL https://docs.cloud.google.com/compute/docs/ accelerator-optimized-machines#a2-ultra-vms. [369] Google LLC. General-purpose machine types: N4 standard (n4-standard-16), 2026. URL https://docs.cloud.google.com/compute/docs/general-purpose-machines# n4-standard. [370] Janez Demšar. Statistical comparisons of classifiers over multiple data sets. Journal of Machine learning research, 7(Jan):1–30, 2006. [371] Sture Holm. A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics, pages 65–70, 1979. [372] Zbyněk Šidák. Rectangular confidence regions for the means of multivariate normal distributions. Journal of the American statistical association, 62(318):626–633, 1967. [373] Milton Friedman. The use of ranks to avoid the assumption of normality implicit in the analysis of variance. Journal of the american statistical association, 32(200):675–701, 1937. [374] Frank Wilcoxon. Individual comparisons by ranking methods. Biometrics bulletin, 1(6):80–83, 1945. [375] William H Kruskal and W Allen Wallis. Use of ranks in one-criterion variance analysis. Journal of the American statistical Association, 47(260):583–621, 1952. 30
[376] Henry B Mann and Donald R Whitney. On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics, pages 50–60, 1947. [377] Charles Spearman. The proof and measurement of association between two things., 1961. [378] Eugène Berta, David Holzmüller, Michael I Jordan, and Francis Bach. Structured matrix scaling for multi-class calibration. arXiv preprint arXiv:2511.03685, 2025. [379] Eugène Berta, David Holzmüller, Michael I Jordan, and Francis Bach. Rethinking early stopping: Refine, then calibrate. arXiv preprint arXiv:2501.19195, 2025.
31
Appendices Table of Contents. A Background
33
A.1 Time-series Forecasting Tasks vs. Temporal Tabular Tasks . . . . . . . . . . . . . B Dataset Curation
33 34
B.1 Towards Automated Data Curation? . . . . . . . . . . . . . . . . . . . . . . . . .
34
B.2 Details on Dataset Selection Criteria . . . . . . . . . . . . . . . . . . . . . . . . .
34
B.3 Details on Dataset Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
35
B.4 Details on Curation Outcome . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
36
B.5 BeyondArena Datasets Overview . . . . . . . . . . . . . . . . . . . . . . . . . . .
36
C Experimental Setup
41
C.1 Time Limit Impact . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
41
C.2 Additional Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
41
D Results
43
D.1 Inference Time Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
43
D.2 BeyondArena Leaderboard as a Table . . . . . . . . . . . . . . . . . . . . . . . .
43
D.3 TabArena-v0.1 Datasets vs. BeyondArena Datasets in the Same Scope . . . . . . .
43
E Statistical Analysis
46
E.1 Test 1: Are the methods really different from each other? . . . . . . . . . . . . . .
46
E.2 Test 2: At what training-set size does using a TFM stop being a reasonable choice?
46
E.3 Test 3: How much do tuning and ensembling help non-TFM models? . . . . . . . .
47
E.4 Test 4: What dataset properties predict that GBDTs will beat TFMs? . . . . . . . .
48
E.5 Test 5: Which method should a practitioner pick, given their data regime? . . . . .
48
F Ablations
51
F.1
(Outer Splits, A.1) Using Fewer Splits with BeyondArena-Core . . . . . . . . . . .
51
F.2
(Outer Splits, A.2) IID Splits for non-IID Grouped Data . . . . . . . . . . . . . . .
52
F.3
(Inner Splits, B.1) Using 8-fold Cross-Validation for Tiny Data . . . . . . . . . . .
52
F.4
(Inner Splits, B.2) Using IID Splits for non-IID Data . . . . . . . . . . . . . . . .
52
F.5
(Grouped Data Preprocessing, C.1a/b) Disabling Preprocessing for Grouped Data. .
54
F.6
(Text Data Preprocessing, C.2a/b) TF-IDF for Text Encoding. . . . . . . . . . . . .
54
F.7
(Post-processing, D) Using Probability Calibration for Log-loss. . . . . . . . . . .
54
G Per Dataset Results
57
H NeurIPS Paper Checklist
93
32
A
Background
A.1
Time-series Forecasting Tasks vs. Temporal Tabular Tasks
We argue that there is a fundamental difference between the definitions of time-series forecasting and temporal tabular data tasks, yielding distinct validation requirements. In short: time-series forecasting tasks predict for the future; temporal tabular tasks predict in the future – thus, when validating a model, it cannot or can use data from the future depending on the task. To illustrate, assume, we have a time index t ∈ T and are given training data Dpast = ({xs ∈ X : s ≤ t}, {ys ∈ Y : s ≤ t}) with the goal to learn to predict Y . In time-series forecasting tasks with a forecast horizon h, we fit on Dpast and predict Yfuture = { ys ∈ Y : t < s ≤ t + h } – at time point t. In tabular temporal data tasks, we fit on Dpast at time point t and predict Yfuture = { ys ∈ Y : t < s } at time point s, with s ∈ T, s > t, using Dpast and Xfuture = { xs }. That is, in temporal tabular tasks, we fit a model at time point t and deploy it to predict at time point s using data from s. Moreover, such tasks do not have a forecast horizon but a refit horizon, which determines the time until we refit the model on a new Dpast . While the many nuances and language differences between the two fields of structured data are out of scope here, the key difference for us lies in how the task affects the data. When naively using data from time-series forecasting tasks, one would violate the assumptions underlying the data and its associated validation process. For example, if we were to use a forecasting dataset and validate a model as if we were predicting at time points s instead of t, the validation would be unrepresentative of the dataset’s task. Moreover, it becomes questionable what the validation would tell us; for many forecasting tasks, one could just read off the target variable (e.g., the stock price) at time points s. When this is the original intended task, we argue that it is a temporal tabular task and not a forecasting problem. For us, the distinction between time-series forecasting data and temporal tabular data is important during data curation. If we naively use time-series forecasting data for temporal tabular regression, we ignore the real-world context of the data. Moreover, the validation procedure would not accurately reflect how well a real-world tabular temporal model performs, but instead estimates how well the model can solve a task, with at best questionable relevance in reality. As we aim to curate data and benchmark models such that the results are as representative of the real-world as possible, we create and use this distinction between data from time-series forecasting and temporal tabular task.
33
B
Dataset Curation
B.1
Towards Automated Data Curation?
We investigated (semi-)automated procedures to aid dataset selection, but found that most of our selection criteria cannot be faithfully applied – not due to a lack of tooling, but to a significant lack of documented, structured metadata required for curation. In most cases, the data and its context from the data repository do not contain the necessary information to make decisions. Moreover, asking agentic LLMs to determine the metadata led to hallucination and faulty judgment. Instead, subjective human judgment and collaborative discussion were required to obtain the metadata for our final decisions. We found agentic tooling most useful for documenting metadata created by humans. In particular, we used Agent Skills [217] via Claude Code [218] to fill metadata templates from human input. Nevertheless, the future of high-quality automated data curation is promising, and we strongly believe that human-curated metadata and standardized schema, like in DataFoundry, will enable us to codify data curation. Therefore, we also share our agent skill markdown file as part of DataFoundry. B.2
Details on Dataset Selection Criteria
Below, we provide a detailed description of each criterion used in BeyondArena. Our selection criteria are based on the work by Erickson et al. [11]. Moreover, our criteria involve subjective human interpretation. Thus, we publicly share our curation insights and decisions for each dataset at https://tabarena.github.io/data-foundry/. Unique Dataset: We ensure datasets in BeyondArena are unique to avoid biasing our results towards a particular distribution. To do so, we ensure that each dataset in BeyondArena has a unique original data source. When we started evaluating new datasets from OpenML and Kaggle, we quickly realized that the only way to identify duplicates was to determine the original source of each dataset, because many datasets were re-uploaded to OpenML and Kaggle under different names and without proper attribution to the original source. In particular, on Kaggle, we noticed that gamified incentives (such as upvotes and medals) might have even prompted users to avoid making their data look like duplicates, so that new, original contributions result in a larger reward. At the same time, Kaggle offers a discussion board for datasets, which enables the community to call out bad behavior and data quality issues. As a result, given a dataset from any data repository, our first priority is to determine its original source, or if it’s an original contribution. In many cases, this is trivial; in others, it requires prolonged investigation, including comparing data statistics and comparing the dataset to popular datasets from the same domain. When the source remains unclear even after a thorough investigation, we leave it to the curator team’s subjective judgment to assess the dataset’s authenticity and uniqueness. No Few-shot Prediction Tasks: We exclude datasets that have fewer than 100 train samples after creating the train-test split. Such data, often called few-shot prediction tasks, require a fundamentally different validation protocol, as data scarcity creates unique challenges, such as the inability to split the data for cross-validation. At the same time, it enables unique solutions to be highly competitive, such as using LLMs for tabular machine learning [111, 123, 127, 131, 134]. While our benchmark focuses on 1 000 000 training samples, we do not exclude larger datasets; instead, we sub-sample them. IID and non-IID Tabular Datasets: We include any tabular dataset whose original task requires a random, temporal, or grouped split. We exclude datasets from time-series forecasting tasks because benchmarking such data in a representative way for their real-world context would require a fundamentally different validation protocol than data from temporal tabular tasks (see Appendix A.1). We leave benchmarking tabular machine learning models for time-series forecasting tasks to benchmarks from the time-series forecasting community [81–84]. Predictive Machine Learning Tasks for Classification or Regression: We exclude datasets that do not originally stem from a predictive machine learning task for classification or regression. Following TabArena, we exclude data from scientific discovery tasks (e.g., survey data or 34
non-predictive tables). For BeyondArena, we also exclude data from other tabular-adjacent tasks, in particular, click-through rate prediction tasks [219–221] and ranking prediction or information retrieval tasks used in recommender systems [158, 222–224]. Like for data from time-series forecasting tasks, datasets from such tasks require specific validation protocols to represent their real-world context appropriately. Furthermore, such tasks have their own benchmarking communities. Representative for Real-world Applications of Tabular Machine learning: BeyondArena aims to understand how well tabular machine learning models would perform when practitioners use them in real-world applications. Consequently, we exclude datasets that represent tasks for which tabular machine learning models would not be used. Moreover, we exclude datasets that do not represent real-world applications or cannot be preprocessed to be representative. Practically speaking, this means we exclude datasets that (A) are from non-tabular modalities, where modality-specific models are clearly superior; (B) do not stem from a real random distribution; (C) are trivial; (D) have irreversible data quality issues; (E) and for which we did not find sufficient information to make an informed decision. (A) We exclude datasets from non-tabular modalities, where modality-specific models are clearly superior. However, unlike TabArena, we determine whether modality-specific models are superior for each dataset rather than excluding all datasets from non-tabular modalities. Consequently, we allow the inclusion of vectorized image, text, audio, or time-series data if tabular machine learning models are competitive with modality-specific solutions. We investigate recent benchmarks related to the application of the dataset or introductory paper to determine if modality-specific solutions are clearly superior. Even if we’re not domain experts, we believe our curation team is sufficiently well-versed in interpreting academic papers to determine whether tabular models are competitive with modality-specific solutions. (B) We exclude datasets that do not stem from a real random distribution. In line with the TabArena definition, we exclude artificial data and data generated by a deterministic function. Unlike TabArena, we believe that simulated data (e.g., the Higgs bosons dataset [225]) represent real random distributions from important real-world applications. Nevertheless, we do not include such datasets in BeyondArena, as such tasks usually have dedicated domain-specific benchmarks [226, 227]. (C) We exclude datasets that are trivial because practitioners would not use machine learning to solve trivial datasets. A dataset is deemed trivial if all models, without tuning, consistently achieve the same better-than-random score. That is, there is no variance, or all models solve the dataset perfectly (error of 0). We utilize the performance of models from BeyondArena. (D) We exclude datasets that have irreversible data quality issues. Many datasets shared online are already preprocessed; some have been processed in ways that make them unusable for benchmarking. Thus, we follow TabArena and exclude datasets where irreversible preprocessing leaks the target or test feature distribution (e.g., PCA-transformed data). (E) We exclude datasets for which we did not find sufficient information to determine whether the datasets would pass our selection criteria. Datasets shared online often come with only limited information about their source and significant ambiguity about their authenticity. In such cases, if a prolonged investigation failed to identify any source information, it was up to the curation team to decide whether the data should be used for benchmarking. Ethically Unambiguous Tasks: We exclude datasets with tasks that pose ethical concerns. As part of BeyondArena, we extend the scope of ethical concerns to include cases in which data subjects or data creators request that the data not be used for machine learning research, such as the Pima Indian Diabetes Dataset (PIDD) [228]. B.3
Details on Dataset Processing
All datasets underwent a processing procedure designed to create consistency and add reproducibility. Across datasets, we observed some recurring processing patterns, which we share below. Case-by-case details and comments for the processing can be found in DataFoundry. General: Many datasets come with unique sample identifiers. We drop all uninformative sample identifiers. Informative sample identifiers, such as a time index, were kept and processed to represent their original meaning (e.g., time) where possible. In many cases, sample identifiers were an artifact of data storage. For similar reasons, many datasets contained missing values represented by proxy values 35
(e.g., 999, -1). We converted these to explicit missing values (NA) whenever such encoding could be reliably inferred from the data description and task context. For numerical target variables with strong skewness and heavy tails (e.g., housing prices), logarithmic scaling was considered on a case-by-case basis. Dataset names were standardized using snake_case naming conventions. We always shuffle the sample order for both IID and grouped data to avoid methods exploiting implicit order leakage. For temporal data, samples were sorted chronologically by their time index. Feature Type Assignments (Dtypes): We transformed features with object or string representations to categorical or string type. We used a categorical type when the number of possible values was fixed and finite (as inferred from the data description or task context). Otherwise, we used a string type. We converted date features to standardized YYYY-MM-DD datetime representations whenever possible. For date features originally encoded numerically, we reconstructed as much of the date information as possible. All other features were encoded as numerical types. Creating Temporal Tasks and Splits: For temporal prediction tasks, we manually define the prediction horizon and the associated test time points. Then, we verified that each feature was valid to use, i.e., the information from the feature would have been available at prediction time, thereby preventing target leakage from future information. We filtered the samples and features as needed to avoid any temporal leakage Datasets were additionally analyzed for grouped temporal structure, such as repeated observations from the same entities over time (e.g., multiple transactions from a single customer). In such cases, using a group split was carefully considered during split construction. We ensure that the first split of a temporal task is always the one with the most recent time test point. Thus, it is also the split with the most training data, and we deem it the most representative split. We order all other splits by descending time test points. B.4
Details on Curation Outcome
We investigate additional characteristics of our curated dataset. In Figure B.1, we show how many datasets we have per domain, and in Figure B.2, we show how the dataset scales (in terms of number of rows and columns) varies across domains. Our datasets cover a broad range of domains with a large focus on some of the most common applied disciplines such as healthcare, marketing and finance. Interestingly, we noticed that medical and healthcare datasets are generally smaller, whereas finance datasets are several orders of magnitude larger on average. In Figure B.3, we show the distribution of tabular datasets with text in our collection. Our text datasets span a wide range of scales, with multiple text columns and varying average text lengths. In Figure B.4, we show the distribution of the cardinality across all categorical features. We see a clear long-tail for high cardinality, making up 14% of all categorical columns. In Figure B.5, we investigate how the scale and dimensionality of our datasets change based on the task type and problem type. We observe that temporal data tends to have larger average sizes, while multiclass classification often stems from smaller problems. In Figure B.6, we show the distribution of time horizons for temporal tasks. In most cases, the time horizon is days or months. Two temporal datasets have only a time index, decoupled from dates. Both used higher step sizes for the prediction horizon. TabArena Reinvestigation. We re-investigated all accepted datasets from TabArena during our data curation process and filtered 2 out of the 51 datasets. We removed the anneal dataset because it is trivial (all models achieved perfect accuracy). Moreover, we removed the diabetes dataset due to ethical concerns, already raised by Radin [228] in 2017. Finally, while ingesting the datasets into DataFoundry, we fixed several minor issues, including dropping constant columns, ensuring correct feature types, and using log scaling for price prediction tasks when appropriate. B.5
BeyondArena Datasets Overview
Table B.1 provides a per-dataset overview of all datasets used in BeyondArena.
36
38
Medical & Healthcare 33
Business & Marketing 18
Finance 13
Biology & Life Sciences 10
Technology & Internet 8
Industry & Manufacturing 7
Chemistry & Material Science 5
Environmental Science & Climate Physics & Astronomy
3
Education
3 3
Insurance 1
Social Science 0
5
10
15
20
25
30
35
40
Number of Datasets
Figure B.1: Dataset Domain Breakdown. We use the domain categories from TALENT [112] and indicate how many out of our 142 datasets stem from each domain.
Medical & Healthcare (n=38) Business & Marketing (n=33) Finance (n=18) Biology & Life Sciences (n=13) Technology & Internet (n=10)
Domain
Industry & Manufacturing (n=8) Chemistry & Material Science (n=7) Environmental Science & Climate (n=5) Physics & Astronomy (n=3) Education (n=3) Insurance (n=3) Social Science (n=1) 102
103
105
104
106
Number of Rows
101
102
103
104
Number of Columns
Figure B.2: Dataset Scale Per Domain. We show the number of rows and columns for each dataset binned by their domain.
20
100
15
10 10
5
Text Length (Average Character Count)
Number of Text Columns
25
0 103
105
104
106
Number of Rows
Figure B.3: Characteristics Tabular Data with Text Features. We show the number of rows and the number of text columns. The hue highlights the average text length. We have an even mix of short and long text datasets, spanning several orders of magnitude in sample size and a diverse number of text columns.
37
Number of Categorical Columns (n=1432)
High cardinality (>50) n=201 (14.0%)
600 500 400 300 200 100 0 3
10
100
1k
10k
100k
Cardinality
Figure B.4: Distribution Cardinality of Categorical. The distribution of the cardinality of all categorical columns across all datasets in our dataset collection. Columns
Rows
106
106
105
105
104
104
Count
Count
Rows
103
Columns
103 102
102
101
101
IID (n=103)
Grouped (n=18)
Binary Classification (n=73)
Temporal (n=21)
Task Type
Regression (n=44)
Multiclass Classification (n=25)
Problem Type
Figure B.5: (Left) Data Scales Per Task. The row and column count for each dataset, binned by task type. We see a trend towards larger temporal datasets. (Right) Data Scales Per Problem. Datasets binned by problem type show that multiclass classification problems are smaller on average, while binary classification and regression are similarly distributed in size.
days: 9 (43%, avg=11.9) months: 6 (29%, avg=3.7)
N = 21
years: 4 (19%, avg=1.0) steps: 2 (10%, avg=5351.0)
Figure B.6: Distribution of Time Horizons for Temporal Dataset. We show the count of time horizon units (days, months, years, steps) and the average number of values per unit (e.g. 11.9 days or 3.7 months). The time horizon indicates the time until we refit or get new labeled data. 38
Table B.1: Per-dataset metadata for BeyondArena benchmark, sorted by number of rows (N ). Columns: N rows, d columns before preprocessing, C classes (regression: —), problem type (Binary classification / Multiclass / Regression ), task type (IID / Temporal / Grouped). Domain abbreviations: M & H = Medical & Healthcare; B & M = Business & Marketing; B & L = Biology & Life Sciences; T & I = Technology & Internet; I & M = Industry & Manufacturing; C & M = Chemistry & Material Science; E & C = Environmental Science & Climate; P & A = Physics & Astronomy. Dataset
Domain
Source
Year
Age
N
d
C
Prob.
Task
hepatitis_survival_prediction [229] cirrhosis_patient_survival_prediction [230] clock_protein_toxicity [231] pancreatic_cancer_mouse_detection [232] lung_cancer_epithelial_genexp [233] parkinsons_biomedical_voice_measurements [234] lung_cancer [235] audiology_diagnosis [236] heart_disease_va_long_beach [237] forensic_glass_identification [238] early_stage_diabetes_risk_prediction [239] body_density_prediction [240] ljubljana_breast_cancer [241] heart_disease_hungary [237] heart_failure_followup_survival [242] ljubljana_primary_tumor [243] heart_disease_cleveland [237] biomechanical_orthopaedic_prediction [244] gallstone_disease [245] prostate_cancer_detection [246] ecoli_proteins [247] horse_colic_survival [248] blood_tests_drink_prediction [249] eryhemato_squamous_disease [250] dementia_prediction [251] south_africa_coronary_heart_disease [252] obesity_estimation [253] telemonitoring_parkinsons_biomedical_voice_measurements [254] forest_fires [255] qsar_aquatic_toxicity [256] micro_mass [257] indian_liver_patient_dataset [258] drug_induced_autoimmunity_prediction [259] hepatitis_c_prediction [260] biogeographical_ancestry_prediction [58, 261, 262] student_portuguese_performance [263] credit_approval [264] blood_transfusion [265] regensburg_pediatric_appendicitis [266] mutual_funds_india [267] qsar_fish_toxicity [268] tour_travels_churn [269] credit_g [270] maternal_health_risk [271] concrete_compressive_strength [272] qsar_biodeg [273] mice_protein_trisomy_discriminant [274] garments_worker_productivity [275] asp_potassco_classification [276, 277] wine_world_cost [278] healthcare_insurance_expenses [279] website_phishing [280] fitness_club [281] airfoil_self_noise [282] fiat_500 [283] mic [284] bad_customer_detection [285] cardiotocography [286] marketing_campaign [287] coffee_rating_prediction [288] hazelnut_spread_contaminant_detection [289] seismic_bumps [290] iranian_churn [291] sat11_hand_algo_runtime [216, 277, 292] splice [293] thyroid_discordant [264] bioresponse [294] hiva_agnostic [295] mercedes_benz_greener_manufacturing [296] predict_students_dropout_and_academic_success [297] santander_transaction_value [298]
M&H M&H B&L M&H M&H M&H M&H M&H M&H C&M M&H M&H M&H M&H M&H M&H M&H M&H M&H M&H B&L B&L M&H M&H M&H M&H M&H M&H E&C B&L B&L M&H M&H M&H B&L Education Finance M&H M&H Finance B&L B&M Finance M&H C&M B&L B&L I&M T&I B&M M&H T&I B&M P&A T&I M&H B&M M&H B&M B&M B&L E&C B&M T&I B&L M&H B&L C&M I&M Education Finance
UCI UCI UCI Other GOV Website UCI Other UCI UCI UCI UCI Kaggle UCI UCI UCI UCI UCI UCI UCI Other UCI UCI UCI UCI Other Kaggle UCI UCI UCI UCI UCI UCI UCI UCI GitHub UCI UCI UCI Other Kaggle UCI Kaggle UCI UCI UCI UCI UCI UCI ASlib Kaggle Kaggle UCI Kaggle UCI Kaggle UCI Kaggle UCI Kaggle Kaggle OpenML UCI UCI ASlib UCI UCI Kaggle Other Kaggle UCI Kaggle
1981 1984 2021 2003 2006 2007 2001 1987 1989 1987 2019 1985 1988 1989 2020 1987 1989 2011 2023 2002 1996 1989 1996 1997 2010 1983 2019 2007 2008 2014 2013 2012 2025 2018 2025 2008 1987 2008 2021 2023 2015 2021 1994 2020 1998 2013 2015 2020 2014 2023 2023 2014 2023 2014 2020 2020 2020 2010 2020 2023 2020 2013 2011 2011 1991 1986 2012 2007 2017 2021 2018
45 42 5 23 20 19 25 39 37 39 7 41 38 37 6 39 37 15 3 24 30 37 30 29 16 43 7 19 18 12 13 14 1 8 1 18 39 18 5 3 11 5 32 6 28 13 11 6 12 3 3 12 3 12 6 6 6 16 6 3 6 13 15 15 35 40 14 19 9 5 8
155 161 171 181 187 195 197 199 200 214 251 252 286 294 299 302 303 310 319 322 327 344 345 366 370 462 498 502 517 546 571 583 597 608 635 649 690 748 763 793 908 954 1,000 1,014 1,030 1,054 1,080 1,197 1,212 1,279 1,338 1,353 1,500 1,503 1,538 1,699 1,723 2,126 2,240 2,369 2,400 2,584 2,850 2,960 3,190 3,711 3,751 3,845 4,204 4,424 4,447
19 17 1,117 6,771 22,215 23 12,600 68 13 9 16 13 9 13 12 17 13 6 38 15,154 6 20 5 34 8 9 14 19 12 8 1,082 10 177 12 104 30 15 4 51 12 6 6 20 6 8 41 76 15 136 14 6 9 6 5 7 111 13 22 25 12 30 15 13 169 60 26 1,776 1,518 371 36 540
2 – 2 2 2 2 4 3 2 6 2 – 2 2 2 11 2 3 2 2 5 3 – 6 3 2 – – – – 20 2 2 4 10 – 2 2 2 – – 2 2 3 – 2 8 – 11 – – 3 2 – – 8 2 3 2 – 2 2 2 – 3 2 2 2 – 3 –
Binary Reg Binary Binary Binary Binary Multi Multi Binary Multi Binary Reg Binary Binary Binary Multi Binary Multi Binary Binary Multi Multi Reg Multi Multi Binary Reg Reg Reg Reg Multi Binary Binary Multi Multi Reg Binary Binary Binary Reg Reg Binary Binary Multi Reg Binary Multi Reg Multi Reg Reg Multi Binary Reg Reg Multi Binary Multi Binary Reg Binary Binary Binary Reg Multi Binary Binary Binary Reg Multi Reg
IID IID IID Grouped IID Grouped IID IID IID IID IID IID IID IID IID IID IID IID IID IID IID IID IID IID Grouped IID IID Grouped IID IID Grouped IID IID IID IID IID IID IID IID IID IID IID IID IID IID IID Grouped Temporal Grouped IID IID IID IID IID IID IID IID Grouped IID Temporal IID IID IID Grouped IID IID IID IID Temporal IID IID
39
Table B.1: (continued) Per-dataset metadata for BeyondArena benchmark. Dataset
Domain
Source
Year
Age
N
d
C
Prob.
Task
churn [299] homeq_default_prediction [300] qsar_tid_11 [301] polish_companies_bankruptcy [302] wine_quality [303] musk [304] taiwanese_bankruptcy_prediction [305] naticusdroid_android_permissions_dataset [306] coil_2000 [307] bank_customer_churn [308] immoscout_german_house_prices [309, 310] heloc [311] jm1 [312] ghanas_indigenous_intel [313] ecommerce_shipping [314] video_game_fps_prediction [315] online_shoppers_purchasing_intention_dataset [316] in_vehicle_coupon_recommendation [317] miami_housing [318] emscad [319] early_learning_predictors [320] hr_analytics [321] houses [322] superconductivity [323] sberbank_housing_market_forecasting [324] credit_card_clients_default [325] amazon_employee_access [326] california_house_prices_2020 [327] bank_marketing [328] food_delivery_time [329] physiochemical_protein [330] anes_voting_2026 [331] kdd_cup_09_appetency [332] diamonds [333] otto_group_product_classification_challenge [334] labour_inspection_compliance [335] video_transcoding_time_prediction [336] santander_customer_satisfaction [337] diabetes_130_us [338] kick [339] aps_failure [340] sdss_17 [341] hotel_booking_demand [342] 5g_energy_consumption [343] sepsis_survival_minimal_clinical_records [344] sf_permit_time [345] wids_diabetes_mellitus [346] customer_satisfaction_in_airline [347] pva_revenue_prediction_kddcup98 [348] give_me_some_credit [349] acquire_valued_shoppers_challenge [350] kickstarter [351] allstate_claims_severity [352] santander_customer_transaction_prediction [353] homesite_quote_conversion [354] home_credit_default_risk [355] covertype [356] ieee_fraud_detection [357] porto_seguro [358] rossmann_store_sales [359] lending_club_1m [360] home_credit_default_stability_1m [324] consumer_complaints_1m [361] sepsis_prediction_1m [362] amex_non_iid_1m [363] delivery_eta_1m [364] cooking_time_1m [364] climate_model_weather_forecasting_1m [364] maps_router_eta_1m [364] mercari_price_suggestion_1m [365] electric_motor_temperature_prediction [366]
T&I B&M C&M Finance C&M C&M Finance T&I B&M B&M B&M Finance T&I E&C B&M T&I B&M B&M Finance B&M Education B&M B&M P&A B&M Finance B&M B&M Finance B&M C&M Social Science B&M B&M B&M I&M T&I B&M M&H B&M I&M P&A B&M T&I M&H B&M M&H B&L B&M Finance B&M B&M Insurance Finance Insurance Finance E&C Finance Insurance B&M Finance Finance Finance M&H Finance I&M I&M E&C I&M B&M I&M
OpenML Other OpenML UCI UCI UCI UCI UCI UCI Kaggle Kaggle Kaggle OpenML Zindi Kaggle OpenML UCI UCI Kaggle Other Other Kaggle Other UCI Kaggle UCI Kaggle Kaggle UCI Kaggle UCI Other Other Other Kaggle Other UCI Kaggle UCI Kaggle UCI Kaggle Other HuggingFace UCI GOV Website Kaggle Kaggle Other Kaggle Kaggle Other Kaggle Kaggle Kaggle Kaggle UCI Kaggle Kaggle Kaggle Kaggle Kaggle GOV Website Other Kaggle Kaggle Kaggle Kaggle Kaggle Kaggle Kaggle
2005 2016 2015 2010 2009 1994 2009 2021 2000 2020 2019 2021 2004 2025 2021 2020 2017 2017 2016 2014 2023 2021 1990 2018 2017 2009 2010 2021 2012 2023 2013 2026 2008 2015 2015 2019 2015 2016 2014 2011 2016 2022 2019 2023 2020 2025 2021 2023 1997 2011 2014 2025 2016 2019 2015 2018 1998 2019 2017 2015 2018 2024 2025 2019 2022 2024 2024 2024 2024 2018 2021
21 10 11 16 17 32 17 5 26 6 7 5 22 1 5 6 9 9 10 12 3 5 36 8 9 17 16 5 14 3 13 0 18 11 11 7 11 10 12 15 10 4 7 3 6 1 5 3 29 15 12 1 10 7 11 8 28 7 9 11 8 2 1 7 4 2 2 2 2 8 5
5,000 5,708 5,741 5,790 6,497 6,598 6,819 7,491 9,822 10,000 10,317 10,459 10,885 10,928 10,999 12,288 12,330 12,684 13,776 17,460 18,874 19,158 19,675 21,263 27,195 30,000 32,769 41,528 45,211 45,451 45,730 48,587 50,000 53,940 61,878 63,634 68,784 71,080 71,518 72,983 76,000 78,053 81,418 92,629 110,204 116,954 127,358 129,880 144,095 150,000 160,057 187,118 188,317 200,000 260,753 307,507 512,625 590,540 595,206 844,392 1,064,751 1,224,927 1,226,140 1,228,686 1,249,605 1,250,000 1,250,000 1,250,000 1,250,000 1,250,000 1,296,316
19 12 1,024 64 12 166 92 85 85 10 23 23 21 10 10 38 17 24 15 17 743 12 8 81 386 23 9 41 13 9 9 318 212 9 93 376 18 307 44 32 170 11 31 20 3 37 181 21 477 10 111 15 130 600 295 504 13 435 37 15 96 711 12 42 189 225 196 100 988 6 109
2 2 – 2 – 2 2 2 2 2 – 2 2 4 2 – 2 2 – 2 – 2 – – – 2 2 – 2 – – 2 2 – 9 2 – 2 2 2 2 3 2 – 2 – 2 2 2 2 2 2 – 2 2 2 3 2 2 – 2 2 3 2 2 – – – – – –
Binary Binary Reg Binary Reg Binary Binary Binary Binary Binary Reg Binary Binary Multi Binary Reg Binary Binary Reg Binary Reg Binary Reg Reg Reg Binary Binary Reg Binary Reg Reg Binary Binary Reg Multi Binary Reg Binary Binary Binary Binary Multi Binary Reg Binary Reg Binary Binary Binary Binary Binary Binary Reg Binary Binary Binary Multi Binary Binary Reg Binary Binary Multi Binary Binary Reg Reg Reg Reg Reg Reg
IID IID IID IID IID Grouped IID IID IID IID IID IID IID Temporal IID Grouped IID IID IID IID Grouped IID IID IID Temporal IID IID Temporal IID IID IID Temporal IID IID IID IID Grouped IID IID Temporal IID IID Temporal Grouped IID Temporal IID IID IID IID Temporal Temporal IID IID IID IID Grouped Temporal IID Temporal Temporal Temporal Temporal Grouped Grouped Temporal Temporal Temporal Temporal IID Grouped
40
Model runtime distribution Cumulative training runtime (s)
104
Training runtime (s)
108
CatBoost TabM RealMLP XGBoost LightGBM Linear RandomForest ExtraTrees TabDPT TabICLv2 TabPFN-2.6 14400 Seconds
103
102
101
Cumulative model runtime distribution
107
106
105
104
103
102
101
0.0
0.2
0.4
0.6
0.8
Proportion of Model Configurations
1.0
0.0
0.2
0.4
0.6
0.8
Proportion of Model Configurations
1.0
Figure C.1: Training runtime distribution across hyperparameter configurations. Each point shows the training runtime, in seconds, for a hyperparameter configuration evaluated on one train split of a dataset, aggregated across all models. The x-axis shows the cumulative proportion of configurations, and the y-axis shows training time in seconds. The vast majority of configurations finish well below the 4-hour time limit: only ∼1.72% exceed 1 hour, ∼0.01% exceed 3.5 hours, and ∼0.003% (31 out of 785,208) exceed the 4-hour limit.
C
Experimental Setup
C.1
Time Limit Impact
In our experiments, we limit the evaluation of a single configuration on a single train split of a dataset to 4 hours. Thus, a model must complete training across all inner folds within 4 hours; otherwise, training is gracefully stopped early. We use a 4-hour limit instead of the 1-hour limit used in TabArena because our benchmark includes larger datasets. Figure C.1 shows the training runtime distribution across all hyperparameter configurations and models, visualizing the proportion of configurations (x-axis) that required a given training time in seconds (y-axis). We find that the time limit is rarely reached. Only ∼1.72% of jobs exceeded 1 hour, ∼0.01% exceeded 3.5 hours, and only ∼0.003% of jobs, corresponding to 31 out of 785,208, exceeded the 4-hour time limit. C.2
Additional Details
Below, we provide more details on specific parts of our experimental setup. Validation Splits & Insufficient Samples per Class. To avoid errors when evaluating validation metrics due to missing classes, we dynamically reduce the number of folds to match the minimum number of samples per class when a dataset has an insufficient number of minority-class samples for cross-validation (i.e., fewer than 5 or 8). Date Encoding We use the DatetimeEncoder from Skrub [201] to convert datetime features into 10 numerical features that represent values such as the year or weekday, as well as a spline-based periodic encoding of the timestamp. Text Encoding We use Qwen3-Embedding-8B to transform each text feature into a 4096dimensional vector, from which we take the first 32 dimensions using Qwen’s supported minimal Matryoshka representation learning [205] slice. We also add caching, so we only need to compute the text encoding once. 41
Grouped Preprocessing We add preprocessing for grouped non-IID data, executed at the end of the model-agnostic preprocessing. For label-per-group datasets, we replace the group index with a transductive 50-dimensional group-encoding. Per group: we (a) sort all samples by a group time index, when available; (b) aggregate all features across samples (mean/std/min/max/last for numeric, count/last/nunique for non-numeric); (c) select the top 50 aggregations by variance when fitting the preprocessing; and (d) join the top 50 aggregated features back to all original samples. At test time, when given a new group as input, we replace the group’s index with these 50 aggregations computed over the group’s samples. This preprocessing assumes a predictive machine learning task in which all samples from a group are available when predicting. This is the case for all label-per-group datasets in BeyondArena. Compute Hardware. We run all our experiments on the Google Cloud Platform (GCP) [207]. We selected virtual machines (VMs) from GCP’s offering that might be representative of the hardware of an average practitioner, while being reasonably affordable for running large-scale experiments. For datasets with fewer than 250k samples, we ran MLPs (TabM, RealMLP) on a2-highgpu-1g [367] VMs with an NVIDIA A100 GPU with 40GB VRAM, 12 vCPUs, and 85 GB RAM. For MLPs on larger datasets, and for any TFM (TabDPT, TabPFN-2.6, TabICLv2), we use a2-ultragpu-1g [368] VMs with an NVIDIA A100 GPU with 80 GB VRAM, 12 vCPUs, and 170 GB RAM. For all other models, we use n4-standard-16 [369] VMs with 16 vCPUs and 64 GB RAM.
42
Improvability vs Inference Time
30.0
Default Tuned Tuned + Ens. Pareto Front
Improvability (%)
27.5 25.0 22.5
Linear RandomForest ExtraTrees TabDPT XGBoost LightGBM CatBoost TabM RealMLP TabICLv2 TabPFN-2.6
20.0 17.5 15.0
Op
10.0
tim al
12.5 101
100
Inference time per 1K samples (s) (median) Figure D.1: Improvability vs. Inference Time Pareto Front. We show the Improvability and inference time of models with the default hyperparameters, with tuning, and with tuning and post-hoc ensembling. The default performance of TFMs equals their in-context learning performance.
D
Results
D.1
Inference Time Overview
We show the Pareto front of Improvability vs. inference time in Figure D.1. We observe that CatBoost dominates the inference time Pareto front, while RealMLP is competitive by default. TabICLv2 and TabPFN-2.6 both require significantly longer inference time while achieving 2.5% Improvement over a tuned CatBoost. A tuned RealMLP incurs an even larger inference time. D.2
BeyondArena Leaderboard as a Table
Table D.1 shows the per-model performance across all datasets. D.3
TabArena-v0.1 Datasets vs. BeyondArena Datasets in the Same Scope
To assess whether the newly added datasets that were in the same scope as the TabArena datasets pose a bigger challenge, we compare ELO on two dataset subsets: (1) the 49 accepted TabArena datasets, (2) 20 datasets which fall in the selection criteria of TabArena (IID, 500 − 250, 000 samples). In Figure D.2, we see that while tabular foundation models (TFMs) dominate on the TabArena datasets, they are outperformed by RealMLP on the new datasets. While Elo performance is stable for non-foundation models, all three evaluated TFMs degrade substantially on the new datasets. This can be explained by the fact that several of the new dataset additions are at the upper end of TabArena’s sample-size limits, are often high-dimensional, and frequently contain high-cardinality categorical features. Overall, this confirms that even datasets within the scope of previous benchmarks can help measure future improvements of TFM on more challenging datasets.
43
Table D.1: BeyondArena Leaderboard. We show default (D), tuned (T), and tuned + ensembled (T+E) performances. Results for TabPFN-2.6 and TabDPT are imputed for datasets with more than 100k samples. The best three values in columns are highlighted with gold, silver, and bronze colors. For Elo values, we also indicate their approximate 95% confidence intervals obtained through bootstrapping. Model
Elo (↑)
RealMLP (T+E) 1282−26,+30 TabPFN-2.6 (D) 1224−35,+40 TabM (T+E) 1210−24,+28 CatBoost (T+E) 1208−19,+27 TabICLv2 (D) 1205−42,+39 RealMLP (T) 1202−23,+31 CatBoost (T) 1189−21,+27 LightGBM (T+E) 1187−22,+26 TabM (T) 1168−27,+25 LightGBM (T) 1149−23,+23 CatBoost (D) 1139−20,+29 XGBoost (T+E) 1138−22,+29 XGBoost (T) 1112−22,+26 TabM (D) 1107−24,+24 RealMLP (D) 1056−25,+23 TabDPT (D) 1052−44,+34 ExtraTrees (T+E) 1008−27,+23 XGBoost (D) 1000−25,+29 LightGBM (D) 991−18,+23 ExtraTrees (T) 971−29,+26 RandomForest (T+E) 963−30,+23 RandomForest (T) 946−26,+24 Linear (T+E) 890−38,+28 RandomForest (D) 876−32,+25 ExtraTrees (D) 864−40,+34 Linear (T) 855−41,+32 Linear (D) 793−50,+38
Improva- Avg. Harm. #wins (↑) Train time Predict time bility (↓) rank (↓) mean per 1K [s] per 1K [s] rank (↓) 10.3% 9.6% 11.9% 12.3% 9.8% 11.9% 12.5% 12.7% 12.7% 13.3% 13.8% 14.6% 14.9% 14.5% 15.8% 17.3% 18.1% 17.6% 17.7% 19.0% 19.3% 19.7% 26.3% 22.2% 23.1% 27.2% 29.7%
7.2 8.9 9.3 9.4 9.5 9.6 10.0 10.0 10.7 11.3 11.6 11.7 12.5 12.7 14.4 14.5 16.0 16.3 16.6 17.2 17.5 18.0 19.7 20.1 20.4 20.7 22.2
44
3.8 3.0 5.1 5.4 2.7 5.1 5.7 5.9 5.8 6.9 7.3 7.2 8.1 8.0 8.8 5.7 11.0 11.8 12.2 11.8 12.9 13.3 11.5 15.5 14.1 12.6 13.6
9.9 24.4 4.9 6.6 33.3 6.9 5.8 5.7 5.1 3.0 2.8 2.9 2.6 2.5 2.5 9.2 1.1 1.0 1.4 1.1 0.9 0.8 2.0 0.6 1.2 1.8 2.0
1316.51 25.26 1072.09 425.71 11.41 1316.51 425.71 163.08 1072.09 163.08 14.38 271.48 271.48 29.89 43.31 107.79 113.01 6.59 3.47 113.01 143.88 143.88 204.03 9.76 6.31 204.03 6.44
14.54 2.39 6.78 1.10 2.42 1.68 0.20 2.42 1.01 0.61 0.17 6.83 1.68 0.98 1.55 11.22 1.62 1.35 0.23 0.34 1.47 0.41 0.90 0.69 0.72 0.21 0.22
ELO (BeyondArena datasets)
1600 TabICLv2 TabPFN-2.6 REALMLP CAT TABM GBM XGB TABDPT XT RF (tuned) LR RF (default)
1500 1400 1300 1200 1100 1000 1000
1100
1200
1300
1400
1500
ELO (TabArena-v0.1 datasets)
1600
Figure D.2: Elo parity scatterplot comparing the 49 accepted TabArena datasets with 20 BeyondArena datasets within the same scope. The dashed diagonal indicates equal performance relative to a default random forest across the two dataset groups.
45
E
Statistical Analysis
We follow the standard practice for comparing multiple learning algorithms across many datasets [370]. All tests are based on normalized error enorm ∈ [0, 1]: on each dataset, methods are scored from best (0) to worst (1) among the 27 methods (11 default, 8 tuned, 8 tuned + ensembled), then averaged across cross-validation folds. Normalized errors let us work on a common scale, even though the benchmark spans heterogeneous tasks with their respective metrics: ROC AUC, log-loss, and RMSE. Whenever a test produces multiple comparisons, we control the family-wise error rate with the Holm–Šidák correction [371, 372]. Holm makes no assumption that the individual tests are independent, which matters here because comparisons across the same 142 datasets yield p-values that are inherently correlated rather than independent. We report effect sizes alongside corrected p-values. E.1
Test 1: Are the methods really different from each other?
The first question to settle is whether the leaderboard ranking is genuinely informative or whether it could plausibly arise from random fluctuations across the 142 datasets. We answer this with a Friedman test [373]. The Friedman test ranks the 27 methods within each dataset and asks whether the resulting rank distributions across datasets are consistent with all methods being equivalent. Because ranks are computed within each dataset, the test naturally accounts for the fact that the same 142 datasets are scored under every method. The Friedman test rejects the null hypothesis: the observed test statistic is χ226 = 1374.9, giving p < 10−273 . To go beyond this global statement and identify which specific pairs of methods differ, we follow up with pairwise Wilcoxon signed-rank tests [374] over all 27 2 = 351 method pairs, with Holm N B N correction. For each pair of methods (A, B) with normalized errors {eA i }i=1 and {ei }i=1 on the A B N = 142 datasets, let di = ei − ei be the per-dataset paired difference. The Wilcoxon signed-rank test drops the magnitudes of the di and works only with the ranks of their absolute values |di |, keeping their signs. The hypotheses are: H0 : the differences di are symmetrically distributed around zero; H1 : the differences are not symmetric around zero (two-sided). We reject H0 at level α = 0.05 when the corresponding two-sided p-value falls below 0.05. Of those 351 pairs, 260 (74.6%) show performance differences that survive Holm correction. This sharpens the global Friedman result: not only is the leaderboard ranking real, but most individual pairs of methods can be reliably distinguished from one another. E.2
Test 2: At what training-set size does using a TFM stop being a reasonable choice?
We split the 142 datasets into the four scale categories used throughout the paper — tiny (100-1k rows), small (1k-10k), medium (10k-100k), and large (≥ 105 ) — and compare the distribution of TFM normalized error across them with a Kruskal-Wallis test [375]. In Test E.1, every dataset is scored under all 27 methods, so the methods can be ranked within each dataset and Friedman aggregates those within-dataset rankings across the 142 rows. Here, by contrast, each dataset belongs to exactly one scale category: the four scale groups are disjoint sets of datasets so Wilcoxon and Friedman are not suitable. Formally, let gi ∈ {1, 2, 3, 4} denote the scale category of dataset i ∈ {1, . . . , N } with N = 142, and let ei be the TFM normalized error on dataset i. We rank the N datasets globally on ei from best (1) to worst (N ) across all four groups combined, and let R̄g denote the mean rank within scale group g, with ng the number of datasets in that group. The hypotheses are: H0 : all four scale groups have the same distribution of TFM normalized error; i.e., dataset size has no systematic effect on TFM error. H1 : at least one group is stochastically different from the others; i.e., TFM error tends to be larger (or smaller) on datasets of some sizes than others. 46
The Kruskal-Wallis test statistic is 2 4 X 12 N +1 H = ng R̄g − , N (N + 1) g=1 2 which under H0 is approximately χ2 -distributed with k − 1 = 3 degrees of freedom. We reject H0 at significance level α = 0.05 when the corresponding p-value falls below 0.05. The Kruskal-Wallis statistic is H = 30.6 on 3 degrees of freedom (p = 1.0 × 10−6 ), so the four scale categories do not share the same distribution of TFM errors. Kruskal-Wallis tells us that the four scale groups are not identically distributed, but it does not say which pairs differ or in which direction. To localize where the change actually happens, we follow up with the two-sample analogue of Kruskal-Wallis: p airwise Mann-Whitney U tests [376] between every pair of scale categories, with Holm correction over the 42 = 6 comparisons (Table E.1). The pattern is clear. Tiny, small, and medium scales are statistically indistinguishable from each other (every pairwise padj > 0.5): a TFM does not perform worse on a 50k-row dataset than on a 500-row one. This pattern breaks at the large-scale boundary: every comparison involving large datasets is highly significant (padj < 0.001) and the rank-biserial effect sizes are large (r between 0.68 and 0.79). The takeaway is that TFM degradation happens at ∼ 100, 000 rows. Table E.1: Pairwise Mann-Whitney U tests for TFM normalized error across dataset scales, Holmcorrected over six comparisons. ēA and ēB are the mean TFM normalized errors in scales A and B. The rank-biserial correlation r measures the size of the gap on a [−1, +1] scale: a positive r means scale B has the higher (worse) error. The three rows comparing large against the other scales are statistically significant after correction (in bold); the other three are not. For example: the third row says that comparing tiny (nA = 52) against large (nB = 22), the mean TFM error rises from 0.183 to 0.528, with a large rank-biserial effect size of r = +0.71 (TFM errors on tiny datasets are systematically smaller than on large datasets), and the corresponding Holm-corrected p-value is below 0.001.
E.3
Scale A
Scale B
tiny (n = 52) tiny tiny small small medium
small (n = 38) medium (n = 30) large (n = 22) medium large large
nA
ēA
ēB
r
padj
0.183 0.183 0.183 0.135 0.135 0.212
0.135 0.212 0.528 0.212 0.528 0.528
−0.11 +0.06 +0.71 +0.20 +0.79 +0.68
0.769 0.769 < 0.001 *** 0.505 < 0.001 *** < 0.001 ***
Test 3: How much do tuning and ensembling help non-TFM models?
Traditional models seem to require tuning and ensembling to reach their full potential, while TFMs rely on in-context learning by default. We now test each of the eight traditional algorithms in BeyondArena to determine whether tuning and post hoc ensembling are actually required. For each algorithm, across all 142 datasets, we have three matched observations: default error, tuned error, and tuned-plus-ensembled error. The matched structure is the right setting for a Wilcoxon signed-rank test. We run two such tests per algorithm — (default → tuned) and (tuned → tuned+ensemble) — for a total of 16 tests jointly Holm-corrected. Formally, fix a traditional algorithm a ∈ {CatBoost, LightGBM, Linear, RandomForest, RealMLP, tun ens TabM, XGBoost, ExtraTrees}, and let edef i , ei , ei denote its normalized errors on dataset i ∈ {1, . . . , 142} when run with default hyperparameters, with tuned hyperparameters, and with tunedplus-ensembled hyperparameters respectively. We test two paired hypotheses per algorithm: (tune)
: the per-dataset differences di
(ens)
: the per-dataset differences di
H0
H0
(tune)
tun = edef i − ei are symmetric around 0;
(ens)
ens = etun i − ei are symmetric around 0.
Under each H0 , the matched pairs convey no systematic improvement; under the corresponding H1 , one pipeline variant is consistently better than the other. As in Test E.1, we compute the signed-rank statistic on the differences. We reject H0 at level α = 0.05 when the corresponding two-sided p-value 47
falls below 0.05 after Holm correction across all 2 × 8 = 16 tests jointly. Every single one of the 16 tests is significant at padj < 0.001 (Table E.2). Both HPO and ensembling consistently improve traditional models across the entire benchmark. RealMLP shows the largest jump from tuning to ensembling (r = 0.87), while LightGBM shows the largest jump from the default to its tuned version (r = 0.89). Table E.2: Wilcoxon signed-rank tests for the impact of tuning and post-hoc ensembling, computed independently for each of the eight non-TFM algorithms (n = 142 datasets per algorithm). The first three columns show the mean normalized error of each pipeline variant. The last two columns give the rank-biserial effect size r for the two transitions; a positive r indicates improvement. Every cell marked *** is significant at padj < 0.001 after Holm correction over all 16 tests. The entry shown in bold per column is the largest effect. Effect size r
Mean normalized error
E.4
Algorithm
Default
Tuned
+Ensemble
Default→Tuned
Tuned→Ens.
CatBoost LightGBM Linear RandomForest RealMLP TabM XGBoost ExtraTrees
0.204 0.343 0.713 0.497 0.271 0.237 0.346 0.517
0.174 0.195 0.614 0.393 0.166 0.189 0.236 0.370
0.167 0.176 0.571 0.379 0.121 0.173 0.224 0.341
0.48 *** 0.89 *** 0.53 *** 0.57 *** 0.80 *** 0.46 *** 0.84 *** 0.65 ***
0.51 *** 0.63 *** 0.77 *** 0.40 *** 0.87 *** 0.79 *** 0.58 *** 0.65 ***
Test 4: What dataset properties predict that GBDTs will beat TFMs?
For each of the 142 datasets, we summarize the head-to-head outcome with a single signed quantity, ∆ = ē⋆TFM − ē⋆GBDT , where ē⋆ is the minimum normalized error achieved by any member of that family on the dataset. Negative ∆ means the best TFM beat the best GBDT; positive ∆ means the opposite. We then correlate ∆ with a set of meta-features of our datasets using Spearman’s rank correlation ρ [377]. The eight meta-feature tests are jointly Holm-corrected. Two meta-features stand out (Table E.3): dataset size and the number of high-cardinality categorical columns. The maximum cardinality of any single categorical column also predicts a GBDT advantage. Missing values, while a weaker signal, also push the balance toward GBDTs. Table E.3: Spearman rank correlation between dataset meta-features and the per-dataset TFM-vsGBDT advantage ∆ = eTFM − eGBDT . A positive ρ means the meta-feature predicts a larger GBDT advantage. The sample size n varies because a few features (e.g. minority-class share, max categorical cardinality) are undefined on regression tasks or on datasets with no categorical columns. We used Holm correction over all features. Meta-feature num_rows num_high_cardinality_cats max_categorical_cardinality missing_value_fraction num_cols_after_preproc age minority_class_pct num_cols num_text_cols
E.5
n
ρ
142 142 105 142 142 142 98 142 142
+0.603 +0.466 +0.413 +0.286 +0.212 −0.242 −0.211 +0.197 +0.051
praw −14
< 10 < 10−8 < 10−4 0.0006 0.011 0.004 0.038 0.019 0.545
padj −13
< 10 < 10−7 < 10−3 0.003 0.045 0.019 0.075 0.056 0.545
Sig. *** *** *** ** * * ns ns ns
Test 5: Which method should a practitioner pick, given their data regime?
Test E.1 established that there is a real global ranking across all 27 methods that is not noise, and Tests E.2–E.4 answered specific sub-questions about how dataset size, tuning, and dataset metafeatures affect performance. None of these directly answers the question a practitioner actually faces: given a dataset with a specific combination of properties, which method should I deploy? A global 48
ranking averaged across all 142 datasets may not match the ranking inside any one regime — TFMs lead on average but lose to neural networks on large or non-IID data, for instance. To turn the benchmark into actionable model-selection guidance, we run Test E.1’s procedure separately within each of the 11 sub-benchmarks. The subsets correspond to one Friedman test per axis level along four orthogonal axes: task type (IID, temporal, grouped — 3 subsets), dataset scale (tiny, small, medium, large — 4 subsets), dimensionality (low-dim, high-dim — 2 subsets), and special feature types (text, high-cardinality — 2 subsets), for 11 sub-benchmarks in total. Each subset gets its own Friedman test on the methods restricted to its datasets, followed by Holm-corrected pairwise Wilcoxon tests within that subset. For each subset s with Ns datasets, the Friedman test asks whether the 27 methods are equivalent within that subset: (s)
H0
: all 27 methods have the same expected within-subset rank;
(s) H1 : at least one method has a different expected rank within subset s. (s)
When H0 is rejected, we localize which method pairs differ by running pairwise Wilcoxon signedrank tests within the subset. For each pair of methods (A, B) with paired per-dataset normalized-error B differences di = eA i − ei on the Ns datasets in subset s: (s,A,B)
H0
: the differences di are symmetrically distributed around zero;
(s,A,B) H1 : the differences are not symmetric around zero (two-sided).
Holm correction is applied jointly over all
27 2
= 351 pairs within each subset.
A coherent picture emerges across the sub-benchmarks (Table E.4): TFMs are the top-1 method on IID, tiny/small, and low-dimensional data. Tuned and ensembled neural networks (RealMLP) lead on temporal, grouped, medium, large, high-dimensional, and text subsets. Tuned GBDTs (CatBoost) lead on high-cardinality categorical data. The grouped subset is the one place where this narrative deserves a caveat: although RealMLP is the top-ranked method, only 5.1% of pairwise comparisons within the grouped subset survive Holm correction, meaning the ranking among the remaining methods is uncertain.
49
Table E.4: The Friedman test is significant (p < 0.001) in every one of the 11 subsets: the per-subset rankings are real. The fraction of pairwise comparisons that survive Holm correction within a subset (column “% sig. pairs”) varies considerably, from 75.5% down to 5.1%. A high fraction means the subset is large and homogeneous enough that most methods can be reliably ordered against each other; a low fraction (the grouped subset, with only 18 datasets) means that even though some method is clearly best, the ranking among the others is too noisy. We report the top method and its family in each subset; for subsets with low sig. pairs, only the top method itself should be read as a robust recommendation. “Top-1 method” is the method with the smallest mean Friedman rank in the subset, with [D] denoting default-mode and [T+E] tuned-plus-ensembled. χ2
p
Sig. pairs
% sig.
103 21 18
1086.8 345.4 147.4
< 10−3 < 10−3 < 10−3
265/351 154/351 18/351
Dataset scale scale=tiny scale=small scale=medium scale=large
52 38 30 22
408.9 500.2 481.9 398.5
< 10−3 < 10−3 < 10−3 < 10−3
Dimensionality dim=low-dim dim=high-dim
91 51
927.5 568.4
Feature type feature=text feature=high-card.
16 30
229.2 502.5
Subset
n
Task type task=random task=temporal task=grouped
Top-1 method
Family
75.5 43.9 5.1
TabICLv2 (default) RealMLP (t+ens) RealMLP (t+ens)
TFM Neural Neural
163/351 214/351 196/351 196/351
46.4 61.0 55.8 55.8
TabICLv2 (default) TabPFN-2.6 (default) RealMLP (t+ens) RealMLP (t+ens)
TFM TFM Neural Neural
< 10−3 < 10−3
254/351 207/351
72.4 59.0
TabICLv2 (default) RealMLP (t+ens)
TFM Neural
< 10−3 < 10−3
85/351 208/351
24.2 59.3
RealMLP (t+ens) CAT (t+ens)
Neural Tree
50
F
Ablations
We ablate BeyondArena to determine the validity of our results. We summarize the results in Table 1 in the main paper. Here, we provide more details on the experimental setup for each ablation. 1. In Appendix F.1, we introduce BeyondArena-Core and show that one can use fewer splits while obtaining similar representative results per dataset. 2. In Appendix F.2, we show that using IID outer splits for non-IID grouped data significantly distorts the results. Benchmarks should use the correct splits for non-IID data. 3. In Appendix F.3, we show that using 5 × 5 cross-validation instead of 8-fold cross-validation for smaller data did lead to better performance for all models, making it a valid pipeline choice. 4. In Appendix F.4, we show that using non-IID inner splits for grouped and temporal data leads to better performance, solidifying the importance of using the appropriate splits per task. 5. In Appendix F.5, we show that our grouped preprocessing leads to big gains, but not for all models. 6. In Appendix F.6, we show that our default text encoding method (Qwen3) is better than a classical TF-IDF method for short text. Yet, it is consistently beaten on large text. Our results remain representative, as the model rankings are highly correlated. 7. In Appendix F.7, we show that probability calibration can significantly boost the performance, mostly for tree-based models. While BeyondArena did not use calibration, future benchmarks should. F.1
(Outer Splits, A.1) Using Fewer Splits with BeyondArena-Core
We use up to 60 splits per dataset. Inspired by TabArena’s Lite subset, which established that benchmark compute costs can be substantially reduced while preserving the aggregate ranking by using only the first split of each dataset, we introduce BeyondArena-Core. We refine the concept of TabArena-Lite to not only preserve aggregate rankings but also to preserve per-dataset rankings, by adjusting the split budget on a per-dataset basis to achieve a per-dataset stability of Φ = 0.8. Formally, for each dataset d with Fd available splits, we define ρd as the mean off-diagonal Spearman correlation of dataset d’s Fd ×Fd split-similarity matrix, i.e., averaged pairwise rank agreement across errors of methods between splits. For each stability target Φ ∈ {0, 0.6, 0.8, 0.9, 0.95, 1} and each dataset d, we determine a per-dataset split budget via Spearman-Brown extrapolation: (1−ρd ) kd (Φ) = min Fd , Φ , kd (1) := Fd . (1) ρd (1−Φ) We then recompute the leaderboard restricted to those tasks and overlay two views on a shared percentage-points axis: (i) aggregate per-method |∆Winrate| against the Φ = 1 reference (Mean and Max curves), and (ii) per-dataset bootstrap winrate jitter at kd (Φ) as a strip plot with cross-dataset medians. The bootstrap is taken with replacement so the Φ = 1 column reflects the irreducible variance of the all-splits estimate rather than collapsing to zero, providing a precision floor against which fidelity loss at smaller Φ can be judged. The dashed reference line marks the per-split mean absolute deviation (MAD) under uniform-random rankings (an upper bound for the k > 1 regime). Figure F.1 shows that the aggregate mean winrate delta is well below 1% regardless of Φ, meaning that even a single split per dataset is sufficient to faithfully represent the overall leaderboard and model rankings, aligning with TabArena’s insights. The per-dataset jitter median, however, grows substantially from Full through Lite: individual datasets’ orderings become noticeably less stable as compute is reduced, even when the benchmark-wide ranking is preserved. In the worst case (Φ = 0), several datasets have a winrate delta of over 20%, which is approaching random noise and thus insufficient for effective dataset-level analysis. By adding splits to satisfy Φ, we efficiently increase compute for the most unstable datasets, resulting in Core (Φ = 0.8) having all datasets with a mean winrate delta below 10% while costing only 60% more than Lite. Benchmark users and creators should use a Core-like subset (Φ = 0.8) when reporting per-dataset results (rank tables, per-dataset wins/losses, dataset-conditioned analyses), which preserves perdataset rank stability while being 5x cheaper than Full; those reporting only aggregate metrics (overall 51
Aggregate winrate fidelity and per-dataset winrate jitter vs Random-noise floor Aggregate Mean | Winrate| Aggregate Max | Winrate| Full ( =1) Core ( =0.8) Lite ( =0) Per-dataset median
25
| Winrate| (% pts)
20
15
10
5
0
=0 9× speedup 142 tasks, 1.4 yrs
=0.6 7× speedup 257 tasks, 1.7 yrs
=0.8 5× speedup 507 tasks, 2.5 yrs
=0.9 3× speedup 975 tasks, 4.0 yrs
Stability target
=0.95 2× speedup 1682 tasks, 6.3 yrs
=1 1× speedup 3722 tasks, 12.3 yrs
Figure F.1: Compute–fidelity trade-off across stability targets Φ. Aggregate per-method |∆Winrate| vs. the Φ = 1 reference (Mean: solid, Max: dotted) overlaid on per-dataset bootstrap winrate jitter at the per-dataset split budget kd (Φ) (Eq. 1); one translucent dot per dataset, black bars mark the cross-dataset median. The dashed line is the per-split MAD under uniform-random rankings (upper bound for k > 1). mean rank, headline winrate) can drop all the way to Lite (Φ = 0) and recover the full speedup (9x faster than Full) with no meaningful loss in fidelity. F.2
(Outer Splits, A.2) IID Splits for non-IID Grouped Data
We investigate the impact on benchmark results when using IID outer train-test splits for two grouped datasets (musk and sat11_hand_algo_runtime). We use the same experimental setup as described in Section 5, but reduce the costs by using a representative subset of models: Linear, ExtraTrees, LightGBM, RealMLP, TabPFN-2.6, and TabICLv2. Figure F.2 shows the results of running all models on the Grouped vs. IID outer splits. For musk, the task becomes trivial under IID splits, with almost all models having an error of 0. At the same time, the variance across folds and the model rankings are heavily distorted. sat11_hand_algo_runtime it looks similiar. While the task does not become trivial, it becomes much simpler. Ranks are distorted again. We conclude, as with temporal data, that the appropriate outer split is crucial for comparing models. F.3
(Inner Splits, B.1) Using 8-fold Cross-Validation for Tiny Data
We investigate the impact of changing the inner splits from 5-repeated 5-fold (5 × 5) CV to 8-fold CV This ablation only uses datasets with ≤ 500 training samples. To reduce the cost of the ablation, we (a) compare on IID datasets with ≤ 100 features, and (b) only run the first three outer folds. We then run the full tuning + ensembling for Linear, ExtraTrees, LightGBM, and RealMLP. We ignore TFMs because they are not affected by the validation protocol; they refit on the full data after cross-validation. Figure F.3 shows that 5 × 5-fold CV performance is better in all cases and gives the largest boost for RealMLP. The rankings are comparable across both protocols. F.4
(Inner Splits, B.2) Using IID Splits for non-IID Data
We used non-IID validation splits for all non-IID datasets in BeyondArena. Here, we investigate how performance changes when using IID splits for validation. We follow the experimental setup from Section 5, using only Linear, ExtraTrees, and LightGBM, and running only the first outer split. 52
Protocol:
Grouped outer split TabICLv2 [D] RealMLP [D] RealMLP [T+E] LightGBM [T+E] LightGBM [T] TabPFN-2.6 [D] RealMLP [T] ExtraTrees [D] ExtraTrees [T+E] ExtraTrees [T] Linear [T] Linear [D] Linear [T+E] LightGBM [D]
TabPFN-2.6 [D] LightGBM [T] ExtraTrees [T+E] LightGBM [T+E] RealMLP [T+E] ExtraTrees [T] ExtraTrees [D] RealMLP [T] RealMLP [D] TabICLv2 [D] LightGBM [D] Linear [T+E] Linear [T] Linear [D]
IID outer split
musk binary classification, label-per-group
0.00
0.05
0.10
0.15
1 - ROC AUC (lower is better)
0.20
0.25 0.0
2.5
5.0
7.5
10.0 12.5 15.0
Mean rank
sat11_hand_algo_runtime regression, label-per-sample
1.5
2.0
2.5
3.0
RMSE (lower is better)
3.5
4.0
0
5
10
15
Mean rank
Figure F.2: Grouped vs. IID Outer Splits. We compare changing the outer splits for two grouped datasets. As a result, (left) the raw metric values and their variances change in a drastic manner, while (right) model ranks are heavily distorted (musk τ = 0.60, sat11_hand_algo_runtime τ = 0.49).
Protocol:
5 × 5-fold CV
8-fold CV
RealMLP [T+E] RealMLP [T] LightGBM [T+E] ExtraTrees [T+E] LightGBM [T] ExtraTrees [T] Linear [T+E] Linear [T] 4
6
8
Mean rank
10
12
Figure F.3: Performance of Cross-validation (CV) Strategies for Tiny Data. We compare Linear, ExtraTrees, LightGBM, and RealMLP with tuning and with post-hoc ensembling on datasets with fewer than 500 samples, when changing the inner splits from 5-repeated 5-fold (5 × 5) CV to 8-fold CV; all other benchmark settings are identical. 5 × 5 CV consistently performs better than 8-fold. Moreover, tuned RealMLP gains a drastic boost, which aligns with it being the most overtuned model in TabArena [11].
53
Protocol:
Non-IID inner splits
Random inner splits
LightGBM [T+E] LightGBM [T] LightGBM [D] ExtraTrees [T+E] ExtraTrees [T] ExtraTrees [D] Linear [T+E] Linear [T] Linear [D] 2
4
6
8
10
Mean rank
12
14
16
Figure F.4: Performance of Non-IID vs IID Inner Splits for Non-IID Data. We compare Linear, ExtraTrees, and LightGBM when using non-IID inner splits (grouped or temporal) as a validation protocol against always using random inner splits. We observe that Linear and LightGBM benefit significantly from non-IID splits, whereas ExtraTrees remains largely unaffected. Moreover, the ranking order is getting mixed: a tuned plus ensembled ExtraTrees model would beat a default LightGBM, only when using random inner splits.
In Figure F.4, we observe that non-IID splits led to better performance in almost all cases, and in such cases, providing a significant boost in performance. F.5
(Grouped Data Preprocessing, C.1a/b) Disabling Preprocessing for Grouped Data.
In BeyondArena, we introduce a default model-agnostic preprocessing step for grouped data that follows industry practice. Here, we examine how the performance would change if we disable this preprocessing. We use the same experimental setup as in Section 5, but only compare the default performance of Linear, ExtraTrees, LightGBM, RealMLP, TabM, TabPFN-2.6, and TabICLv2. Moreover, we restrict the ablation to only using datasets with ≤ 100k training samples. In Figure F.5, we split the ablation by label-per-group (L-P-G) and label-per-sample (L-P-S) datasets, showing that for both, the rankings can change a lot while performance gains with grouped preprocessing are higher on average. For TabPFN-2.6, no preprocessing would have been better. F.6
(Text Data Preprocessing, C.2a/b) TF-IDF for Text Encoding.
BeyondArena contains tabular data with text, which requires encoding for tabular machine learning models. By default, we used Qwen3-Embedding-8B with Matryoshka representation learning (MRL) for text encoding. Here, we ablate using TF-IDF from Skrub with SVD. We run the ablation across all datasets with text, following the experimental setup in Section 5. We use only the first split of each dataset and restrict the runs to the cheaper CPU models: Linear, ExtraTrees, and LightGBM. Figure F.6 shows mixed results. For short text, Qwen3 was the better choice. For long text, TF-IDF was the better choice. In the future, one would need to tune the preprocessing method for each dataset to achieve peak performance. F.7
(Post-processing, D) Using Probability Calibration for Log-loss.
BeyondArena uses log-loss as a metric for multiclass classification. Post hoc calibration can improve the model’s performance when measured by log-loss. Since BeyondArena does not use post hoc 54
Protocol:
Grouped Preprocessing
No Grouped Preprocessing
label-per-group · N = 5 datasets
label-per-sample · N = 9 datasets
TabM [D] TabPFN-2.6 [D] TabICLv2 [D] RealMLP [D] LightGBM [D] Linear [D] ExtraTrees [D] 4
5
6
7
8
Mean rank
9
10
11
12
4
6
8
Mean rank
10
12
Figure F.5: Impact of Preprocessing for Grouped Data. We compare the default performance of Linear, ExtraTrees, LightGBM, RealMLP, TabICLv2, TabPFN-2.6, and TabM with and without the group preprocessing introduced by BeyondArena, split by label-per-group (L-P-G) and label-per-sample (L-P-S) data. We observe that the performance gains depend on the model, while, on average, enabling grouped preprocessing led to better performance. TabPFN-2.6 is a distinguishable outlier compared to all other models, as it benefited the most from not using grouped preprocessing.
Protocol:
Qwen3-Embedding-8B + MRL
TF-IDF + SVD
short text · mean 50 chars · N = 8 datasets
long text · mean > 50 chars · N = 8 datasets
LightGBM [T+E] LightGBM [T] LightGBM [D] ExtraTrees [T+E] ExtraTrees [T] ExtraTrees [D] Linear [T+E] Linear [T] Linear [D] 0.0
2.5
5.0
7.5
10.0
Mean rank
12.5
15.0
17.5
0
5
10
Mean rank
15
20
Figure F.6: LM vs. TF-IDF for Encoding Text Features. We compare the performance of Linear, ExtraTrees, and LightGBM when changing the preprocessing of text features from a language model (LM), in the form of Qwen3-Embedding-8B with Matryoshka representation learning (MRL), to a classical encoding method, in the form of TF-IDF with SVD. Both approaches create a 32-dimensional output vector for each text feature. We split the results across short-text (≤ 50 characters on average across all text cells) and long-text datasets. For short texts, the LM consistently performs better, while for long texts, TF-IDF dominates. In both cases, the model rankings are not affected by the preprocessing choice.
55
Protocol:
No calibration
SMS calibration
4
8
12
TabICLv2 TabPFN-2.6 RealMLP [T+E] TabM [T+E] CatBoost [T+E] LightGBM [T+E] ExtraTrees [T+E] Linear [T+E] TabDPT XGBoost [T+E] RandomForest [T+E] 6
10
Mean rank
14
16
18
Figure F.7: No Calibration vs. Probability Calibration for Multiclass Classification. We compare the performance of all models when adding post hoc probability calibration (via SMS calibration [378]) after tuning and ensembling for multiclass classification. SMS calibration yields significant gains in average rank for several models and alters the model rank order. calibration, we ablate using structured matrix scaling (SMS) for calibration [378], the default of the probmetrics framework [379]. We recompute the log-loss of all methods after SMS using the saved predictions from the main benchmark. In Figure F.7, we show that SMS calibration improved the performance for most methods. In particular, tree-based models improved significantly, while only TabPFN-2.6 and RealMLP performed worse in calibration.
56
G
Per Dataset Results
Performance Per Dataset. We show the average predictive performance per dataset (metric error) with the standard deviation over folds. We show the performance for the default hyperparameter configuration (Default), for the model after tuning (Tuned), and for the ensemble after tuning (Tuned + Ens.). We highlight the best-performing methods with significance on three levels: (1) Green: The best performing method on average; (2) Bold: Methods that are not significantly worse than the best method on average, based on a Wilcoxon Signed-Rank test for paired samples with Holm-Bonferroni correction and α = 0.05. (3) Underlined: Methods that are not significantly worse than the best method in the same pipeline regime (Default, Tuned, or Tuned + Ens.), based on a Wilcoxon Signed-Rank test for paired samples with Holm-Bonferroni correction and α = 0.05. Datasets with only a single split do not receive bold/underline annotations because the significance test degenerates without paired samples.
HPO Pareto Trajectory. Beside each per-dataset table we plot the metric error (y-axis) of each method’s tuned and ensembled configuration against cumulative training time in seconds (x-axis), as we increase the number of HPO trials. Each curve traces the Pareto frontier of validation error versus training-time budget for a single method, so points further down and to the left dominate. Reading the plot together with the table makes it possible to compare the cost (compute budget needed to reach a given error) and the achievable error of each method on that dataset, beyond the single endpoint summary in the table. Each point in the trajectory corresponds from left to right to an ensemble of {1, 2, 5, 10, 15, 20, 26} configurations. The default configuration is always present, with the remaining configurations being randomly selected and the results averaged across 10 seeds.
Tuned + Ens.
0.376 ± 0.042 0.352 ± 0.045 0.357 ± 0.045 0.356 ± 0.042 0.352 ± 0.045 0.353 ± 0.046 0.351 ± 0.042 0.349 ± 0.042 -
0.375 ± 0.041 0.353 ± 0.047 0.356 ± 0.044 0.356 ± 0.043 0.352 ± 0.044 0.353 ± 0.047 0.352 ± 0.045 0.348 ± 0.041 -
Dataset: acquire_valued_shoppers_challeTabICLv2
ExtraTrees CatBoost XGBoost RandomForest LightGBM RealMLP TabM Metadata Samples 160,057 Features 120 Problem binary Splits 5 Type temporal Missingness 0.0% Domain business & marketing Source Kaggle Year 2014
0.3675 0.3650 0.3625 0.3600 0.3575 0.3550 0.3525 0.3500 0.3475
al
Tuned
0.372 ± 0.038 0.365 ± 0.031 0.362 ± 0.035 0.355 ± 0.041 0.354 ± 0.048 0.354 ± 0.048 0.353 ± 0.049 0.353 ± 0.043 0.368 ± 0.041
Op tim
Default
Test Error (1-AUC)
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
101
102
103
Train time (s)
104
105
Figure G.1: acquire_valued_shoppers_challenge: per-method test error (left) and HPO Pareto trajectory (right).
Tuned + Ens.
4.745 ± 0.148 1.928 ± 0.113 1.677 ± 0.121 1.356 ± 0.122 1.530 ± 0.147 1.450 ± 0.126 1.185 ± 0.113 1.216 ± 0.107 -
4.729 ± 0.135 1.874 ± 0.119 1.677 ± 0.123 1.361 ± 0.122 1.528 ± 0.135 1.456 ± 0.127 1.170 ± 0.110 1.195 ± 0.105 -
Dataset: airfoil_self_noise 1.8 1.6 1.4 1.2
RandomForest ExtraTrees LightGBM XGBoost CatBoost TabDPT TabM RealMLP TabPFN-2.6 TabICLv2 Metadata Samples 1,503 Features 5 Problem regression Splits 30 Type IID Missingness 0.0% Domain physics & astronomy Source UCI Year 2014
al
Tuned
5.337 ± 0.152 1.945 ± 0.115 1.898 ± 0.114 1.602 ± 0.102 1.575 ± 0.130 1.574 ± 0.137 1.218 ± 0.101 1.283 ± 0.129 1.203 ± 0.092 1.117 ± 0.096 1.112 ± 0.092
Test Error (RMSE)
Default
Op tim
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
101
102
Train time (s)
103
Figure G.2: airfoil_self_noise: per-method test error (left) and HPO Pareto trajectory (right). 57
0.565 ± 0.001 0.560 ± 0.001 0.558 ± 0.001 0.534 ± 0.001 0.540 ± 0.001 0.538 ± 0.001 0.536 ± 0.001 0.533 ± 0.001 0.542 ± 0.001
0.565 ± 0.001 0.558 ± 0.001 0.558 ± 0.001 0.534 ± 0.001 0.536 ± 0.001 0.538 ± 0.001 0.533 ± 0.001 0.532 ± 0.001 -
0.564 ± 0.001 0.557 ± 0.001 0.557 ± 0.001 0.534 ± 0.001 0.535 ± 0.001 0.536 ± 0.001 0.532 ± 0.001 0.532 ± 0.001 -
Dataset: allstate_claims_severity
RandomForest ExtraTrees TabICLv2 XGBoost LightGBM CatBoost TabM RealMLP Metadata Samples 188,317 Features 130 Problem regression Splits 9 Type IID Missingness 0.0% Domain insurance Source Kaggle Year 2016
0.560 0.555 0.550 0.545 0.540 0.535 tim al
Tuned + Ens.
Test Error (RMSE)
Tuned
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
102
103
Train time (s)
104
105
Figure G.3: allstate_claims_severity: per-method test error (left) and HPO Pareto trajectory (right).
0.149 ± 0.009 0.172 ± 0.006 0.168 ± 0.007 0.120 ± 0.006 0.154 ± 0.007 0.166 ± 0.008 0.153 ± 0.009 0.163 ± 0.007 0.189 ± 0.008 0.152 ± 0.007 0.145 ± 0.008
0.147 ± 0.009 0.172 ± 0.006 0.160 ± 0.007 0.118 ± 0.006 0.145 ± 0.009 0.152 ± 0.011 0.132 ± 0.005 0.159 ± 0.011 -
0.146 ± 0.010 0.172 ± 0.006 0.159 ± 0.007 0.118 ± 0.006 0.141 ± 0.008 0.143 ± 0.008 0.130 ± 0.007 0.151 ± 0.008 -
Dataset: amazon_employee_access TabDPT
0.19
RandomForest ExtraTrees TabPFN-2.6 TabM TabICLv2 XGBoost LightGBM RealMLP CatBoost Metadata Samples 32,769 Features 9 Problem binary Splits 9 Type IID Missingness 0.0% Domain business & marketing Source Kaggle Year 2010
0.18 0.17 0.16 0.15 0.14 0.13 0.12
al
Tuned + Ens.
Test Error (1-AUC)
Tuned
Op tim
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
101
102
104
103
Train time (s)
Figure G.4: amazon_employee_access: per-method test error (left) and HPO Pareto trajectory (right).
Tuned + Ens.
0.061 0.057 0.061 0.056 0.055 0.055 0.055 0.054 -
0.058 0.058 0.060 0.056 0.055 0.055 0.055 0.055 -
Dataset: amex_non_iid_1m 0.062 0.060 0.058 0.056
TabICLv2 ExtraTrees RandomForest CatBoost XGBoost RealMLP LightGBM TabM Metadata Samples 1,249,605 Features 198 Problem binary Splits 1 Type grouped Missingness 8.6% Domain finance Source Kaggle Year 2022
tim
al
Test Error (1-AUC)
Tuned
Op
Default Linear 0.061 RandomForest 0.057 ExtraTrees 0.060 CatBoost 0.057 LightGBM 0.057 XGBoost 0.056 RealMLP 0.058 TabM 0.055 TabDPT TabPFN-2.6 TabICLv2 0.063
103
104
105
Train time (s)
Figure G.5: amex_non_iid_1m: per-method test error (left) and HPO Pareto trajectory (right).
0.077 ± 0.043 0.077 ± 0.046 0.078 ± 0.048 0.067 ± 0.039 0.067 ± 0.038 0.067 ± 0.038 0.067 ± 0.040 0.067 ± 0.039 -
0.16 0.14 0.12 0.10 0.08
TabDPT ExtraTrees RandomForest TabICLv2 TabPFN-2.6 RealMLP TabM LightGBM XGBoost CatBoost Metadata Samples 48,587 Features 318 Problem binary Splits 9 Type temporal Missingness 57.7% Domain social science Source Other Year 2026
al
0.077 ± 0.043 0.085 ± 0.058 0.082 ± 0.053 0.067 ± 0.039 0.068 ± 0.040 0.067 ± 0.038 0.067 ± 0.040 0.067 ± 0.040 -
0.18
Test Error (1-AUC)
0.086 ± 0.044 0.081 ± 0.035 0.096 ± 0.028 0.068 ± 0.041 0.070 ± 0.039 0.067 ± 0.038 0.069 ± 0.041 0.067 ± 0.039 0.184 ± 0.063 0.067 ± 0.040 0.069 ± 0.040
Dataset: anes_voting_2026
Tuned + Ens.
tim
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Tuned
Op
Default
102
103
Train time (s)
104
Figure G.6: anes_voting_2026: per-method test error (left) and HPO Pareto trajectory (right). 58
0.012 ± 0.003 0.009 ± 0.002 0.010 ± 0.002 0.008 ± 0.002 0.008 ± 0.002 0.008 ± 0.002 0.010 ± 0.003 0.008 ± 0.002 0.010 ± 0.003 0.008 ± 0.002 0.006 ± 0.002
0.012 ± 0.003 0.009 ± 0.002 0.010 ± 0.003 0.008 ± 0.002 0.008 ± 0.002 0.008 ± 0.002 0.010 ± 0.002 0.008 ± 0.002 -
0.010 ± 0.002 0.009 ± 0.002 0.009 ± 0.002 0.008 ± 0.002 0.008 ± 0.002 0.008 ± 0.002 0.008 ± 0.002 0.008 ± 0.002 -
Dataset: aps_failure
0.0100
TabDPT RandomForest ExtraTrees LightGBM TabM CatBoost XGBoost RealMLP TabPFN-2.6 TabICLv2 Metadata Samples 76,000 Features 170 Problem binary Splits 9 Type IID Missingness 8.3% Domain industry & manufacturing Source UCI Year 2016
0.0095 0.0090 0.0085 0.0080 0.0075 0.0070 0.0065 tim al
Tuned + Ens.
0.0060
Op
Tuned
Test Error (1-AUC)
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
102
103
Train time (s)
104
Figure G.7: aps_failure: per-method test error (left) and HPO Pareto trajectory (right).
3.815 ± 0.655 2.286 ± 0.149 2.306 ± 0.144 2.231 ± 0.113 2.289 ± 0.078 2.282 ± 0.066 2.546 ± 0.317 2.307 ± 0.183 2.295 ± 0.114 2.187 ± 0.101 2.175 ± 0.097
3.063 ± 0.444 2.287 ± 0.149 2.265 ± 0.146 2.245 ± 0.110 2.267 ± 0.095 2.251 ± 0.079 2.257 ± 0.133 2.251 ± 0.129 -
2.805 ± 0.401 2.279 ± 0.163 2.259 ± 0.139 2.242 ± 0.110 2.265 ± 0.091 2.249 ± 0.078 2.250 ± 0.117 2.242 ± 0.119 -
Dataset: asp_potassco_classification TabDPT
2.55
RandomForest LightGBM ExtraTrees RealMLP XGBoost TabM CatBoost TabPFN-2.6 TabICLv2 Metadata Samples 1,212 Features 136 Problem multiclass Classes 11 Splits 30 Type grouped Missingness 10.9% Domain technology & internet Source ASlib Year 2014
2.50 2.45 2.40 2.35 2.30 2.25 2.20
al
Tuned + Ens.
Test Error (logloss)
Tuned
Op tim
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
101
102
103
Train time (s)
Figure G.8: asp_potassco_classification: per-method test error (left) and HPO Pareto trajectory (right).
0.276 ± 0.084 0.365 ± 0.060 0.361 ± 0.066 0.336 ± 0.065 0.360 ± 0.056 0.349 ± 0.058 0.302 ± 0.066 0.347 ± 0.069 -
0.42
Dataset: audiology_diagnosis
0.40 0.38 0.36 0.34 0.32 0.30 0.28
al
0.293 ± 0.092 0.381 ± 0.086 0.380 ± 0.113 0.338 ± 0.065 0.359 ± 0.057 0.350 ± 0.057 0.310 ± 0.072 0.349 ± 0.070 -
0.44
Test Error (logloss)
0.283 ± 0.086 0.361 ± 0.054 0.361 ± 0.052 0.427 ± 0.059 0.433 ± 0.053 0.368 ± 0.066 0.347 ± 0.052 0.342 ± 0.067 0.402 ± 0.053 0.309 ± 0.070 0.270 ± 0.093
Tuned + Ens.
tim
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Tuned
TabDPT RandomForest ExtraTrees LightGBM XGBoost TabM CatBoost TabPFN-2.6 RealMLP TabICLv2 Metadata Samples 199 Features 68 Problem multiclass Classes 3 Splits 60 Type IID Missingness 2.1% Domain medical & healthcare Source UCI Year 1987
Op
Default
102
Train time (s)
103
Figure G.9: audiology_diagnosis: per-method test error (left) and HPO Pareto trajectory (right).
0.268 ± 0.017 0.269 ± 0.022 0.296 ± 0.022 0.252 ± 0.020 0.273 ± 0.023 0.269 ± 0.026 0.269 ± 0.025 0.261 ± 0.021 0.254 ± 0.021 0.250 ± 0.021 0.252 ± 0.020
0.269 ± 0.017 0.273 ± 0.026 0.279 ± 0.024 0.253 ± 0.022 0.264 ± 0.021 0.253 ± 0.021 0.264 ± 0.023 0.261 ± 0.023 -
0.266 ± 0.018 0.269 ± 0.025 0.277 ± 0.025 0.249 ± 0.020 0.255 ± 0.019 0.252 ± 0.021 0.257 ± 0.020 0.261 ± 0.022 -
Dataset: bad_customer_detection ExtraTrees Test Error (1-AUC)
Tuned + Ens.
RandomForest TabM RealMLP LightGBM TabDPT TabICLv2 XGBoost TabPFN-2.6 CatBoost Metadata Samples 1,723 Features 13 Problem binary Splits 30 Type IID Missingness 0.0% Domain business & marketing Source Kaggle Year 2020
0.29 0.28 0.27 0.26 0.25
tim al
Tuned
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
101
102
Train time (s)
103
Figure G.10: bad_customer_detection: per-method test error (left) and HPO Pareto trajectory (right). 59
0.229 ± 0.005 0.147 ± 0.007 0.146 ± 0.007 0.130 ± 0.006 0.135 ± 0.007 0.136 ± 0.007 0.134 ± 0.006 0.130 ± 0.007 0.134 ± 0.007 0.129 ± 0.006 0.129 ± 0.006
0.228 ± 0.005 0.143 ± 0.007 0.142 ± 0.006 0.129 ± 0.005 0.133 ± 0.006 0.135 ± 0.006 0.131 ± 0.005 0.130 ± 0.007 -
0.228 ± 0.005 0.142 ± 0.007 0.140 ± 0.007 0.129 ± 0.005 0.132 ± 0.006 0.134 ± 0.006 0.130 ± 0.006 0.129 ± 0.006 -
Dataset: bank_customer_churn
0.1475 0.1450 0.1425 0.1400 0.1375 0.1350 0.1325 0.1300
RandomForest ExtraTrees XGBoost TabDPT LightGBM RealMLP TabM CatBoost TabPFN-2.6 TabICLv2 Metadata Samples 10,000 Features 10 Problem binary Splits 9 Type IID Missingness 0.0% Domain business & marketing Source Kaggle Year 2020
tim al
Tuned + Ens.
Op
Tuned
Test Error (1-AUC)
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
101
102
Train time (s)
103
Figure G.11: bank_customer_churn: per-method test error (left) and HPO Pareto trajectory (right).
0.259 ± 0.008 0.241 ± 0.006 0.244 ± 0.006 0.235 ± 0.006 0.237 ± 0.006 0.236 ± 0.006 0.237 ± 0.006 0.237 ± 0.006 -
0.259 ± 0.008 0.240 ± 0.005 0.243 ± 0.006 0.235 ± 0.006 0.236 ± 0.006 0.235 ± 0.006 0.236 ± 0.006 0.236 ± 0.006 -
ExtraTrees RandomForest TabICLv2 TabDPT TabM TabPFN-2.6 RealMLP LightGBM XGBoost CatBoost Metadata Samples 45,211 Features 13 Problem binary Splits 9 Type IID Missingness 0.0% Domain finance Source UCI Year 2012
0.270 0.265 0.260 0.255 0.250 0.245 0.240 0.235
al
0.260 ± 0.008 0.270 ± 0.006 0.275 ± 0.006 0.235 ± 0.006 0.237 ± 0.006 0.238 ± 0.007 0.240 ± 0.007 0.237 ± 0.006 0.238 ± 0.006 0.236 ± 0.006 0.239 ± 0.007
Dataset: bank_marketing
0.275
tim
Tuned + Ens.
Test Error (1-AUC)
Tuned
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
101
102
103
Train time (s)
104
Figure G.12: bank_marketing: per-method test error (left) and HPO Pareto trajectory (right).
1.842 ± 0.075 1.670 ± 0.013 1.684 ± 0.011 1.556 ± 0.028 1.694 ± 0.034 1.634 ± 0.040 1.576 ± 0.039 1.427 ± 0.048 1.511 ± 0.028 1.373 ± 0.045 1.369 ± 0.050
1.647 ± 0.048 1.573 ± 0.037 1.535 ± 0.036 1.527 ± 0.033 1.555 ± 0.034 1.571 ± 0.033 1.427 ± 0.037 1.418 ± 0.046 -
1.647 ± 0.048 1.563 ± 0.032 1.535 ± 0.034 1.529 ± 0.032 1.552 ± 0.033 1.572 ± 0.034 1.420 ± 0.032 1.417 ± 0.045 -
Dataset: biogeographical_ancestry_prediXGBoost 1.70
RandomForest LightGBM ExtraTrees CatBoost TabDPT RealMLP TabM TabPFN-2.6 TabICLv2 Metadata Samples 635 Features 104 Problem multiclass Classes 10 Splits 60 Type IID Missingness 0.0% Domain biology & life sciences Source GitHub Year 2025
1.65 1.60 1.55 1.50 1.45 1.40 al
Tuned + Ens.
Test Error (logloss)
Tuned
Op tim
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
102
Train time (s)
103
Figure G.13: biogeographical_ancestry_prediction: per-method test error (left) and HPO Pareto trajectory (right).
0.340 ± 0.053 0.367 ± 0.037 0.364 ± 0.030 0.349 ± 0.044 0.364 ± 0.041 0.377 ± 0.041 0.332 ± 0.054 0.341 ± 0.048 -
0.41
RandomForest ExtraTrees LightGBM CatBoost TabPFN-2.6 TabM TabDPT TabICLv2 RealMLP Metadata Samples 310 Features 6 Problem multiclass Classes 3 Splits 60 Type IID Missingness 0.0% Domain medical & healthcare Source UCI Year 2011
0.40 0.39 0.38 0.37 0.36 0.35 0.34 al
0.341 ± 0.057 0.368 ± 0.037 0.364 ± 0.030 0.349 ± 0.044 0.367 ± 0.044 0.376 ± 0.042 0.336 ± 0.065 0.341 ± 0.049 -
Dataset: biomechanical_orthopaedic_predXGBoost
0.33
tim
Tuned + Ens.
Test Error (logloss)
Linear 0.373 ± 0.044 RandomForest 0.370 ± 0.029 ExtraTrees 0.411 ± 0.023 CatBoost 0.367 ± 0.043 LightGBM 0.377 ± 0.044 XGBoost 0.403 ± 0.050 RealMLP 0.355 ± 0.080 TabM 0.348 ± 0.036 TabDPT 0.340 ± 0.054 TabPFN-2.6 0.341 ± 0.056 TabICLv2 0.332 ± 0.059
Tuned
Op
Default
102
Train time (s)
103
Figure G.14: biomechanical_orthopaedic_prediction: per-method test error (left) and HPO Pareto trajectory (right). 60
0.211 ± 0.012 0.130 ± 0.012 0.133 ± 0.012 0.130 ± 0.009 0.132 ± 0.010 0.130 ± 0.009 0.142 ± 0.011 0.137 ± 0.010 0.135 ± 0.011 0.125 ± 0.010 0.141 ± 0.011
0.195 ± 0.011 0.130 ± 0.012 0.135 ± 0.012 0.128 ± 0.009 0.130 ± 0.011 0.127 ± 0.011 0.133 ± 0.009 0.127 ± 0.010 -
0.186 ± 0.011 0.129 ± 0.012 0.132 ± 0.011 0.128 ± 0.009 0.129 ± 0.011 0.127 ± 0.011 0.126 ± 0.010 0.127 ± 0.011 -
Dataset: bioresponse
0.1425
TabICLv2 TabDPT ExtraTrees LightGBM RandomForest CatBoost XGBoost TabM RealMLP TabPFN-2.6 Metadata Samples 3,751 Features 1,776 Problem binary Splits 9 Type IID Missingness 0.0% Domain biology & life sciences Source Kaggle Year 2012
0.1400 0.1375 0.1350 0.1325 0.1300 0.1275 0.1250
tim al
Tuned + Ens.
Op
Tuned
Test Error (1-AUC)
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
102
104
103
Train time (s)
Figure G.15: bioresponse: per-method test error (left) and HPO Pareto trajectory (right).
0.683 ± 0.033 0.685 ± 0.031 0.683 ± 0.030 0.686 ± 0.028 0.686 ± 0.030 0.687 ± 0.030 0.681 ± 0.030 0.684 ± 0.031 -
0.683 ± 0.030 0.686 ± 0.031 0.683 ± 0.029 0.685 ± 0.029 0.686 ± 0.030 0.687 ± 0.030 0.681 ± 0.030 0.684 ± 0.031 -
RandomForest LightGBM CatBoost TabM ExtraTrees RealMLP TabPFN-2.6 TabICLv2 TabDPT Metadata Samples 345 Features 5 Problem regression Splits 60 Type IID Missingness 0.0% Domain medical & healthcare Source UCI Year 1996
0.700 0.695 0.690 0.685 0.680 0.675
al
0.681 ± 0.030 0.699 ± 0.032 0.688 ± 0.029 0.684 ± 0.029 0.689 ± 0.029 0.701 ± 0.029 0.681 ± 0.031 0.685 ± 0.030 0.673 ± 0.031 0.678 ± 0.031 0.678 ± 0.031
Dataset: blood_tests_drink_predictionXGBoost
Tuned + Ens.
Test Error (RMSE)
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Tuned
Op tim
Default
102
Train time (s)
103
Figure G.16: blood_tests_drink_prediction: per-method test error (left) and HPO Pareto trajectory (right).
0.252 ± 0.034 0.289 ± 0.032 0.271 ± 0.032 0.256 ± 0.033 0.251 ± 0.033 0.264 ± 0.034 0.250 ± 0.030 0.262 ± 0.033 -
0.247 ± 0.032 0.289 ± 0.032 0.272 ± 0.032 0.257 ± 0.032 0.252 ± 0.034 0.264 ± 0.034 0.249 ± 0.031 0.260 ± 0.032 -
0.30 0.29 0.28 0.27 0.26 0.25
RandomForest ExtraTrees XGBoost TabM CatBoost LightGBM TabPFN-2.6 TabDPT RealMLP TabICLv2 Metadata Samples 748 Features 4 Problem binary Splits 60 Type IID Missingness 0.0% Domain medical & healthcare Source UCI Year 2008
al
0.269 ± 0.032 0.307 ± 0.029 0.304 ± 0.028 0.257 ± 0.032 0.269 ± 0.035 0.285 ± 0.035 0.248 ± 0.032 0.259 ± 0.031 0.251 ± 0.032 0.251 ± 0.032 0.247 ± 0.033
Dataset: blood_transfusion Test Error (1-AUC)
Tuned + Ens.
tim
Tuned
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
102
Train time (s)
103
Figure G.17: blood_transfusion: per-method test error (left) and HPO Pareto trajectory (right).
0.011 ± 0.001 0.011 ± 0.001 0.011 ± 0.001 0.011 ± 0.001 0.011 ± 0.001 0.012 ± 0.001 0.010 ± 0.001 0.011 ± 0.001 0.010 ± 0.001 0.010 ± 0.001 0.010 ± 0.001
0.011 ± 0.001 0.011 ± 0.001 0.011 ± 0.001 0.011 ± 0.001 0.011 ± 0.001 0.012 ± 0.001 0.010 ± 0.001 0.011 ± 0.001 -
0.011 ± 0.001 0.011 ± 0.001 0.011 ± 0.001 0.011 ± 0.001 0.011 ± 0.001 0.012 ± 0.001 0.010 ± 0.001 0.011 ± 0.001 -
0.0118
Dataset: body_density_prediction XGBoost
LightGBM CatBoost RandomForest ExtraTrees TabM TabPFN-2.6 RealMLP TabICLv2 TabDPT Metadata Samples 252 Features 13 Problem regression Splits 60 Type IID Missingness 0.0% Domain medical & healthcare Source Kaggle Year 1985
0.0116 0.0114 0.0112 0.0110 0.0108 0.0106 0.0104
tim al
Tuned + Ens.
Op
Tuned
Test Error (RMSE)
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
102
Train time (s)
103
Figure G.18: body_density_prediction: per-method test error (left) and HPO Pareto trajectory (right). 61
Default
Tuned
Dataset: california_house_prices_2020TabDPT
Tuned + Ens.
RandomForest ExtraTrees TabICLv2 TabM CatBoost XGBoost LightGBM RealMLP TabPFN-2.6 Metadata Samples 41,528 Features 657 Problem regression Splits 3 Type temporal Missingness 15.7% Domain business & marketing Source Kaggle Year 2021
0.240 ± 0.009 0.202 ± 0.009 0.200 ± 0.011 0.195 ± 0.008 0.193 ± 0.010 0.194 ± 0.010 0.193 ± 0.009 0.197 ± 0.011 -
0.30 0.28 0.26 0.24 0.22 0.20
al
0.242 ± 0.009 0.202 ± 0.009 0.201 ± 0.011 0.195 ± 0.008 0.195 ± 0.010 0.195 ± 0.010 0.193 ± 0.009 0.198 ± 0.012 -
tim
0.245 ± 0.012 0.211 ± 0.009 0.202 ± 0.011 0.196 ± 0.008 0.196 ± 0.010 0.198 ± 0.009 0.197 ± 0.009 0.199 ± 0.010 0.322 ± 0.018 0.189 ± 0.010 0.197 ± 0.011
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Test Error (RMSE)
0.32
102
103
Train time (s)
104
Figure G.19: california_house_prices_2020: per-method test error (left) and HPO Pareto trajectory (right).
Default
Tuned
Dataset: cardiotocography
Tuned + Ens. 0.305 ± 0.062 0.241 ± 0.049 0.243 ± 0.046 0.223 ± 0.052 0.225 ± 0.050 0.227 ± 0.052 0.245 ± 0.051 0.252 ± 0.058 -
0.27 0.26 0.25 0.24 0.23 al
0.345 ± 0.073 0.246 ± 0.052 0.245 ± 0.047 0.222 ± 0.052 0.232 ± 0.057 0.231 ± 0.056 0.263 ± 0.054 0.259 ± 0.060 -
tim
0.341 ± 0.065 0.253 ± 0.045 0.281 ± 0.046 0.225 ± 0.052 0.230 ± 0.051 0.231 ± 0.051 0.278 ± 0.062 0.270 ± 0.066 0.258 ± 0.059 0.254 ± 0.070 0.230 ± 0.064
TabDPT TabPFN-2.6 TabM RealMLP ExtraTrees RandomForest TabICLv2 XGBoost LightGBM CatBoost Metadata Samples 2,126 Features 22 Problem multiclass Classes 3 Splits 30 Type grouped Missingness 0.0% Domain medical & healthcare Source UCI Year 2010
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Test Error (logloss)
0.28
101
102
103
Train time (s)
Figure G.20: cardiotocography: per-method test error (left) and HPO Pareto trajectory (right).
0.175 ± 0.006 0.082 ± 0.012 0.082 ± 0.011 0.076 ± 0.005 0.080 ± 0.011 0.077 ± 0.008 0.077 ± 0.010 0.079 ± 0.011 -
0.175 ± 0.006 0.081 ± 0.012 0.080 ± 0.011 0.076 ± 0.005 0.081 ± 0.010 0.079 ± 0.010 0.077 ± 0.009 0.080 ± 0.011 -
RandomForest LightGBM ExtraTrees TabM XGBoost RealMLP CatBoost TabPFN-2.6 TabDPT TabICLv2 Metadata Samples 5,000 Features 19 Problem binary Splits 9 Type IID Missingness 0.0% Domain technology & internet Source OpenML Year 2005
0.0825 0.0800 0.0775 0.0750 0.0725 0.0700 0.0675
al
0.223 ± 0.014 0.083 ± 0.012 0.085 ± 0.010 0.076 ± 0.005 0.083 ± 0.011 0.077 ± 0.008 0.081 ± 0.009 0.078 ± 0.010 0.074 ± 0.012 0.074 ± 0.010 0.067 ± 0.010
Dataset: churn 0.0850
tim
Tuned + Ens.
Op
Tuned
Test Error (1-AUC)
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
101
102
Train time (s)
103
Figure G.21: churn: per-method test error (left) and HPO Pareto trajectory (right).
0.878 ± 0.084 0.867 ± 0.077 0.883 ± 0.079 0.863 ± 0.078 0.869 ± 0.081 0.873 ± 0.081 0.839 ± 0.076 0.834 ± 0.079 -
0.870 ± 0.083 0.867 ± 0.078 0.882 ± 0.079 0.863 ± 0.078 0.868 ± 0.082 0.872 ± 0.080 0.833 ± 0.075 0.833 ± 0.078 -
XGBoost RandomForest LightGBM CatBoost TabICLv2 TabPFN-2.6 TabDPT RealMLP TabM Metadata Samples 161 Features 17 Problem regression Splits 60 Type IID Missingness 13.1% Domain medical & healthcare Source UCI Year 1984
0.90 0.88 0.86 0.84 al
0.905 ± 0.089 0.879 ± 0.082 0.877 ± 0.078 0.875 ± 0.080 0.884 ± 0.080 0.909 ± 0.085 0.840 ± 0.078 0.826 ± 0.079 0.848 ± 0.083 0.849 ± 0.082 0.855 ± 0.086
Dataset: cirrhosis_patient_survival_pre ExtraTrees Test Error (RMSE)
Tuned + Ens.
tim
Tuned
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
102
Train time (s)
103
Figure G.22: cirrhosis_patient_survival_prediction: per-method test error (left) and HPO Pareto trajectory (right). 62
Tuned + Ens.
1.545 1.496 1.519 1.330 1.329 1.341 1.300 1.312 -
1.537 1.494 1.519 1.325 1.321 1.327 1.293 1.312 -
Dataset: climate_model_weather_forecast ExtraTrees
RandomForest TabICLv2 XGBoost CatBoost LightGBM TabM RealMLP
1.45
Metadata Samples 1,250,000 Features 109 Problem regression Splits 1 Type temporal Missingness 0.8% Domain environmental science & climate Source Kaggle Year 2024
1.40 1.35 1.30
al
Test Error (RMSE)
1.50
tim
Linear 1.538 RandomForest 1.496 ExtraTrees 1.526 CatBoost 1.339 LightGBM 1.353 XGBoost 1.341 RealMLP 1.332 TabM 1.340 TabDPT TabPFN-2.6 TabICLv2 1.392
Tuned
Op
Default
103
104
Train time (s)
105
Figure G.23: climate_model_weather_forecasting_1m: per-method test error (left) and HPO Pareto trajectory (right).
0.542 ± 0.069 0.510 ± 0.079 0.498 ± 0.071 0.474 ± 0.067 0.470 ± 0.059 0.461 ± 0.063 0.465 ± 0.063 0.469 ± 0.065 -
0.52 0.50 0.48 0.46
TabDPT RandomForest TabICLv2 ExtraTrees TabPFN-2.6 CatBoost TabM RealMLP LightGBM XGBoost Metadata Samples 171 Features 1,117 Problem binary Splits 60 Type IID Missingness 0.0% Domain biology & life sciences Source UCI Year 2021
al
0.541 ± 0.069 0.511 ± 0.080 0.500 ± 0.071 0.477 ± 0.067 0.472 ± 0.058 0.461 ± 0.066 0.479 ± 0.066 0.468 ± 0.062 -
0.54
Test Error (1-AUC)
0.528 ± 0.075 0.518 ± 0.069 0.509 ± 0.069 0.482 ± 0.065 0.461 ± 0.062 0.477 ± 0.066 0.503 ± 0.069 0.477 ± 0.063 0.544 ± 0.067 0.478 ± 0.068 0.508 ± 0.068
Dataset: clock_protein_toxicity
Tuned + Ens.
tim
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Tuned
Op
Default
102
103
Train time (s)
Figure G.24: clock_protein_toxicity: per-method test error (left) and HPO Pareto trajectory (right).
1.205 ± 0.175 1.212 ± 0.205 1.247 ± 0.257 1.105 ± 0.183 1.136 ± 0.185 1.133 ± 0.196 1.079 ± 0.175 1.104 ± 0.179 1.111 ± 0.140 1.042 ± 0.183 1.080 ± 0.172
1.145 ± 0.150 1.206 ± 0.194 1.230 ± 0.250 1.093 ± 0.165 1.085 ± 0.186 1.091 ± 0.182 1.079 ± 0.184 1.104 ± 0.158 -
1.148 ± 0.187 1.204 ± 0.204 1.229 ± 0.251 1.093 ± 0.168 1.085 ± 0.186 1.091 ± 0.182 1.070 ± 0.190 1.100 ± 0.168 -
1.25
Dataset: coffee_rating_prediction
1.20 1.15 1.10 1.05
ExtraTrees RandomForest TabDPT TabM CatBoost XGBoost LightGBM TabICLv2 RealMLP TabPFN-2.6 Metadata Samples 2,369 Features 207 Problem regression Splits 5 Type temporal Missingness 6.3% Domain business & marketing Source Kaggle Year 2023
al
Tuned + Ens.
Test Error (RMSE)
Tuned
Op tim
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
102
104
103
Train time (s)
Figure G.25: coffee_rating_prediction: per-method test error (left) and HPO Pareto trajectory (right).
0.267 ± 0.020 0.296 ± 0.019 0.296 ± 0.020 0.241 ± 0.021 0.251 ± 0.019 0.242 ± 0.016 0.263 ± 0.021 0.246 ± 0.019 0.276 ± 0.020 0.225 ± 0.021 0.229 ± 0.022
0.264 ± 0.022 0.257 ± 0.021 0.261 ± 0.023 0.240 ± 0.018 0.239 ± 0.019 0.247 ± 0.021 0.244 ± 0.019 0.239 ± 0.022 -
0.264 ± 0.022 0.257 ± 0.020 0.255 ± 0.023 0.240 ± 0.021 0.238 ± 0.020 0.243 ± 0.018 0.242 ± 0.017 0.239 ± 0.021 -
Dataset: coil_2000 0.29
Test Error (1-AUC)
Tuned + Ens. 0.28 0.27 0.26 0.25 0.24 0.23
TabDPT RandomForest ExtraTrees XGBoost RealMLP CatBoost TabM LightGBM TabICLv2 TabPFN-2.6 Metadata Samples 9,822 Features 85 Problem binary Splits 9 Type IID Missingness 0.0% Domain business & marketing Source UCI Year 2000
tim al
Tuned
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
101
102
Train time (s)
103
Figure G.26: coil_2000: per-method test error (left) and HPO Pareto trajectory (right). 63
8.2 ± 0.4 5.0 ± 0.4 4.9 ± 0.4 4.1 ± 0.4 4.2 ± 0.3 4.2 ± 0.3 4.0 ± 0.3 131.7 ± 583.8 -
200
ExtraTrees TabDPT TabM LightGBM XGBoost CatBoost RealMLP TabPFN-2.6 TabICLv2
175 150 125
Metadata Samples 1,030 Features 8 Problem regression Splits 30 Type IID Missingness 0.0% Domain chemistry & material science Source UCI Year 1998
100 75 50 25
al
8.2 ± 0.4 5.0 ± 0.4 4.9 ± 0.4 4.1 ± 0.4 4.2 ± 0.4 4.2 ± 0.3 4.1 ± 0.3 4.2 ± 0.3 -
Dataset: concrete_compressive_strength RandomForest
tim
Tuned + Ens.
Test Error (RMSE)
Linear 8.2 ± 0.4 RandomForest 5.2 ± 0.4 ExtraTrees 5.2 ± 0.4 CatBoost 4.1 ± 0.4 LightGBM 4.4 ± 0.3 XGBoost 4.6 ± 0.4 RealMLP 4.7 ± 0.3 TabM 4.2 ± 0.3 TabDPT 4.2 ± 0.4 TabPFN-2.6 4.0 ± 0.3 TabICLv2 3.9 ± 0.4
Tuned
Op
Default
0
101
102
103
Train time (s)
Figure G.27: concrete_compressive_strength: per-method test error (left) and HPO Pareto trajectory (right).
Tuned + Ens.
0.601 0.437 0.427 0.423 0.430 0.433 0.446 0.426 -
0.565 0.432 0.430 0.425 0.428 0.431 0.435 0.424 -
Dataset: consumer_complaints_1m
0.48 0.47 0.46 0.45 0.44 0.43
TabICLv2 RealMLP RandomForest XGBoost ExtraTrees LightGBM CatBoost TabM Metadata Samples 1,226,140 Features 114 Problem multiclass Classes 3 Splits 1 Type temporal Missingness 16.1% Domain finance Source GOV Website Year 2025
tim al
Test Error (logloss)
Tuned
Op
Default Linear 0.561 RandomForest 0.437 ExtraTrees 0.479 CatBoost 0.431 LightGBM 0.429 XGBoost 0.432 RealMLP 0.446 TabM 0.443 TabDPT TabPFN-2.6 TabICLv2 0.445
104
103
105
Train time (s)
Figure G.28: consumer_complaints_1m: per-method test error (left) and HPO Pareto trajectory (right).
Default
Tuned
Dataset: cooking_time_1m
Tuned + Ens. 0.488 0.486 0.484 0.482 0.480
0.476
al
0.478
TabICLv2 ExtraTrees RandomForest TabM XGBoost RealMLP LightGBM CatBoost Metadata Samples 1,250,000 Features 205 Problem regression Splits 1 Type temporal Missingness 8.1% Domain industry & manufacturing Source Kaggle Year 2024
tim
0.483 0.483 0.485 0.476 0.476 0.477 0.476 0.478 -
Test Error (RMSE)
0.485 0.483 0.485 0.476 0.477 0.478 0.477 0.479 -
Op
Linear 0.487 RandomForest 0.483 ExtraTrees 0.485 CatBoost 0.476 LightGBM 0.481 XGBoost 0.478 RealMLP 0.480 TabM 0.482 TabDPT TabPFN-2.6 TabICLv2 0.488
104
103
105
Train time (s)
Figure G.29: cooking_time_1m: per-method test error (left) and HPO Pareto trajectory (right).
0.819 ± 0.128 0.769 ± 0.056 0.783 ± 0.034 0.739 ± 0.026 0.807 ± 0.016 0.812 ± 0.038 0.776 ± 0.088 0.850 ± 0.102 1.025 ± 0.256
0.805 ± 0.143 0.754 ± 0.049 0.754 ± 0.025 0.715 ± 0.046 0.720 ± 0.058 0.743 ± 0.046 0.733 ± 0.055 0.696 ± 0.109 -
0.805 ± 0.140 0.752 ± 0.053 0.762 ± 0.019 0.717 ± 0.034 0.724 ± 0.051 0.744 ± 0.045 0.712 ± 0.061 0.707 ± 0.098 -
Dataset: covertype 1.00
Test Error (logloss)
Tuned + Ens. 0.95
Metadata Samples 512,625 Features 13 Problem multiclass Classes 3 Splits 3 Type grouped Missingness 0.0% Domain environmental science & climate Source UCI Year 1998
0.90 0.85 0.80 0.75 0.70
TabICLv2 ExtraTrees RandomForest XGBoost LightGBM CatBoost RealMLP TabM
tim al
Tuned
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
101
102
103
Train time (s)
104
Figure G.30: covertype: per-method test error (left) and HPO Pareto trajectory (right). 64
0.075 ± 0.017 0.065 ± 0.015 0.075 ± 0.016 0.064 ± 0.014 0.062 ± 0.014 0.062 ± 0.014 0.068 ± 0.015 0.064 ± 0.015 -
0.074
Dataset: credit_approval
ExtraTrees RealMLP TabDPT RandomForest TabM XGBoost CatBoost LightGBM TabICLv2 TabPFN-2.6 Metadata Samples 690 Features 15 Problem binary Splits 60 Type IID Missingness 0.6% Domain finance Source UCI Year 1987
0.072 0.070 0.068 0.066 0.064 0.062 al
0.076 ± 0.017 0.065 ± 0.015 0.076 ± 0.016 0.065 ± 0.015 0.064 ± 0.015 0.063 ± 0.014 0.069 ± 0.015 0.065 ± 0.015 -
0.076
0.060
tim
0.079 ± 0.016 0.066 ± 0.014 0.074 ± 0.016 0.062 ± 0.014 0.063 ± 0.014 0.067 ± 0.014 0.075 ± 0.016 0.064 ± 0.015 0.066 ± 0.016 0.060 ± 0.014 0.061 ± 0.013
Tuned + Ens.
Test Error (1-AUC)
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Tuned
Op
Default
102
103
Train time (s)
Figure G.31: credit_approval: per-method test error (left) and HPO Pareto trajectory (right).
0.255 ± 0.005 0.219 ± 0.004 0.218 ± 0.003 0.215 ± 0.003 0.214 ± 0.003 0.214 ± 0.003 0.214 ± 0.003 0.212 ± 0.003 -
0.225 0.220 0.215 0.210
RandomForest ExtraTrees TabDPT CatBoost LightGBM XGBoost RealMLP TabM TabPFN-2.6 TabICLv2 Metadata Samples 30,000 Features 23 Problem binary Splits 9 Type IID Missingness 0.0% Domain finance Source UCI Year 2009
al
0.254 ± 0.005 0.219 ± 0.004 0.220 ± 0.002 0.215 ± 0.003 0.215 ± 0.003 0.215 ± 0.003 0.214 ± 0.003 0.211 ± 0.003 -
0.230
tim
0.254 ± 0.005 0.231 ± 0.004 0.229 ± 0.003 0.215 ± 0.003 0.216 ± 0.003 0.217 ± 0.003 0.215 ± 0.003 0.216 ± 0.004 0.216 ± 0.002 0.210 ± 0.003 0.209 ± 0.003
Dataset: credit_card_clients_default
Tuned + Ens.
Test Error (1-AUC)
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Tuned
Op
Default
101
102
Train time (s)
104
103
Figure G.32: credit_card_clients_default: per-method test error (left) and HPO Pareto trajectory (right).
Tuned + Ens.
0.235
0.211 ± 0.026 0.218 ± 0.028 0.221 ± 0.027 0.207 ± 0.028 0.208 ± 0.029 0.208 ± 0.030 0.214 ± 0.029 0.209 ± 0.030 -
0.211 ± 0.027 0.218 ± 0.028 0.220 ± 0.027 0.205 ± 0.028 0.206 ± 0.028 0.208 ± 0.030 0.208 ± 0.028 0.205 ± 0.029 -
0.230
Dataset: credit_g
ExtraTrees TabDPT RandomForest XGBoost RealMLP TabPFN-2.6 LightGBM TabM TabICLv2 CatBoost Metadata Samples 1,000 Features 20 Problem binary Splits 30 Type IID Missingness 0.0% Domain finance Source UCI Year 1994
0.225 0.220 0.215
0.205
al
0.210 tim
Test Error (1-AUC)
Tuned
Op
Default Linear 0.217 ± 0.026 RandomForest 0.215 ± 0.029 ExtraTrees 0.219 ± 0.028 CatBoost 0.210 ± 0.028 LightGBM 0.234 ± 0.025 XGBoost 0.218 ± 0.030 RealMLP 0.222 ± 0.025 TabM 0.207 ± 0.029 TabDPT 0.215 ± 0.028 TabPFN-2.6 0.207 ± 0.026 TabICLv2 0.205 ± 0.026
101
102
Train time (s)
103
Figure G.33: credit_g: per-method test error (left) and HPO Pareto trajectory (right).
0.038 ± 0.002 0.008 ± 0.000 0.008 ± 0.000 0.005 ± 0.000 0.006 ± 0.000 0.006 ± 0.000 0.005 ± 0.000 0.005 ± 0.000 0.006 ± 0.000 0.005 ± 0.000 0.005 ± 0.000
0.038 ± 0.002 0.008 ± 0.000 0.006 ± 0.000 0.005 ± 0.000 0.006 ± 0.000 0.006 ± 0.000 0.005 ± 0.000 0.005 ± 0.000 -
0.038 ± 0.002 0.007 ± 0.000 0.006 ± 0.000 0.005 ± 0.000 0.006 ± 0.000 0.006 ± 0.000 0.005 ± 0.000 0.005 ± 0.000 -
Dataset: customer_satisfaction_in_airliRandomForest 0.0080
ExtraTrees XGBoost LightGBM TabDPT CatBoost RealMLP TabICLv2 TabPFN-2.6 TabM Metadata Samples 129,880 Features 21 Problem binary Splits 9 Type IID Missingness 0.0% Domain biology & life sciences Source Kaggle Year 2023
0.0075 0.0070 0.0065 0.0060 0.0055 0.0050 tim al
Tuned + Ens.
Op
Tuned
Test Error (1-AUC)
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
102
103
Train time (s)
104
Figure G.34: customer_satisfaction_in_airline: per-method test error (left) and HPO Pareto trajectory (right). 65
0.5325
0.874 0.531 0.533 0.515 0.515 0.514 0.514 0.515 -
0.5300 0.5275 0.5250 0.5225 0.5200 0.5175 al
0.5150
ExtraTrees RandomForest TabICLv2 CatBoost TabM LightGBM XGBoost RealMLP Metadata Samples 1,250,000 Features 234 Problem regression Splits 1 Type temporal Missingness 17.2% Domain industry & manufacturing Source Kaggle Year 2024
tim
0.999 0.530 0.533 0.515 0.515 0.516 0.514 0.517 -
Dataset: delivery_eta_1m
Tuned + Ens.
Op
Linear 1.004 RandomForest 0.530 ExtraTrees 0.533 CatBoost 0.516 LightGBM 0.517 XGBoost 0.516 RealMLP 0.517 TabM 0.517 TabDPT TabPFN-2.6 TabICLv2 0.524
Tuned
Test Error (RMSE)
Default
104
103
Train time (s)
105
Figure G.35: delivery_eta_1m: per-method test error (left) and HPO Pareto trajectory (right).
0.624 ± 0.062 0.662 ± 0.053 0.645 ± 0.051 0.646 ± 0.051 0.635 ± 0.057 0.683 ± 0.058 0.619 ± 0.066 0.644 ± 0.068 -
Dataset: dementia_prediction 0.70 0.68 0.66 0.64 0.62
al
Tuned + Ens.
0.625 ± 0.062 0.663 ± 0.058 0.645 ± 0.051 0.645 ± 0.051 0.636 ± 0.061 0.685 ± 0.058 0.624 ± 0.071 0.647 ± 0.072 -
Test Error (logloss)
Tuned
0.719 ± 0.071 0.687 ± 0.048 0.714 ± 0.044 0.668 ± 0.052 0.663 ± 0.053 0.714 ± 0.052 0.649 ± 0.083 0.653 ± 0.066 0.706 ± 0.079 0.671 ± 0.074 0.682 ± 0.077
tim
Default
TabDPT XGBoost TabICLv2 TabPFN-2.6 RandomForest CatBoost ExtraTrees TabM LightGBM RealMLP Metadata Samples 370 Features 8 Problem multiclass Classes 3 Splits 60 Type grouped Missingness 0.7% Domain medical & healthcare Source Other Year 2010
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
102
Train time (s)
103
Figure G.36: dementia_prediction: per-method test error (left) and HPO Pareto trajectory (right).
Tuned + Ens.
0.342 ± 0.002 0.346 ± 0.002 0.350 ± 0.003 0.327 ± 0.003 0.338 ± 0.002 0.330 ± 0.003 0.330 ± 0.004 0.338 ± 0.003 -
0.342 ± 0.002 0.344 ± 0.003 0.347 ± 0.003 0.327 ± 0.003 0.333 ± 0.002 0.329 ± 0.003 0.330 ± 0.002 0.337 ± 0.003 -
Dataset: diabetes_130_us 0.37 0.36 0.35 0.34 0.33
TabDPT ExtraTrees RandomForest TabM TabICLv2 LightGBM TabPFN-2.6 RealMLP XGBoost CatBoost Metadata Samples 71,518 Features 44 Problem binary Splits 9 Type IID Missingness 4.0% Domain medical & healthcare Source UCI Year 2014
al
Tuned
0.352 ± 0.002 0.364 ± 0.002 0.369 ± 0.003 0.328 ± 0.003 0.350 ± 0.003 0.335 ± 0.003 0.341 ± 0.004 0.339 ± 0.003 0.377 ± 0.005 0.332 ± 0.002 0.335 ± 0.002
Test Error (1-AUC)
Default
Op tim
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
101
102
104
103
Train time (s)
Figure G.37: diabetes_130_us: per-method test error (left) and HPO Pareto trajectory (right).
0.198 ± 0.056 0.092 ± 0.002 0.090 ± 0.001 0.084 ± 0.001 0.084 ± 0.001 0.085 ± 0.001 0.081 ± 0.001 0.081 ± 0.000 -
Dataset: diamonds
RandomForest ExtraTrees TabDPT XGBoost LightGBM CatBoost TabPFN-2.6 TabM RealMLP TabICLv2 Metadata Samples 53,940 Features 9 Problem regression Splits 9 Type IID Missingness 0.0% Domain business & marketing Source Other Year 2015
0.092 0.090 0.088 0.086 0.084 0.082 al
Tuned + Ens.
0.202 ± 0.063 0.092 ± 0.002 0.090 ± 0.001 0.084 ± 0.001 0.085 ± 0.001 0.085 ± 0.001 0.082 ± 0.001 0.081 ± 0.001 -
0.080
tim
Tuned
0.195 ± 0.036 0.092 ± 0.002 0.091 ± 0.001 0.084 ± 0.001 0.085 ± 0.001 0.085 ± 0.000 0.086 ± 0.001 0.082 ± 0.001 0.085 ± 0.002 0.082 ± 0.001 0.080 ± 0.001
Test Error (RMSE)
Default
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
101
102
103
Train time (s)
104
Figure G.38: diamonds: per-method test error (left) and HPO Pareto trajectory (right). 66
0.247 ± 0.034 0.174 ± 0.032 0.153 ± 0.032 0.158 ± 0.031 0.169 ± 0.034 0.181 ± 0.035 0.170 ± 0.033 0.170 ± 0.029 -
0.234 ± 0.031 0.173 ± 0.033 0.154 ± 0.032 0.156 ± 0.031 0.170 ± 0.034 0.176 ± 0.033 0.159 ± 0.028 0.167 ± 0.029 -
XGBoost RandomForest LightGBM TabM RealMLP CatBoost ExtraTrees TabPFN-2.6 TabICLv2 Metadata Samples 597 Features 208 Problem binary Splits 60 Type IID Missingness 0.0% Domain medical & healthcare Source UCI Year 2025
0.18 0.17 0.16 0.15 0.14
al
0.245 ± 0.030 0.174 ± 0.032 0.154 ± 0.032 0.164 ± 0.031 0.187 ± 0.035 0.181 ± 0.032 0.188 ± 0.033 0.177 ± 0.033 0.182 ± 0.035 0.150 ± 0.027 0.140 ± 0.028
Dataset: drug_induced_autoimmunity_predTabDPT 0.19
tim
Tuned + Ens.
Op
Tuned
Test Error (1-AUC)
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
102
Train time (s)
103
Figure G.39: drug_induced_autoimmunity_prediction: per-method test error (left) and HPO Pareto trajectory (right).
13.0 ± 0.0 13.0 ± 0.0 13.0 ± 0.0 13.0 ± 0.0 13.0 ± 0.0 13.0 ± 0.0 13.0 ± 0.0 138.0 ± 373.0 -
15000 12500 10000 7500 5000 2500
TabM TabDPT RandomForest ExtraTrees TabPFN-2.6 TabICLv2 XGBoost RealMLP LightGBM CatBoost Metadata Samples 18,874 Features 752 Problem regression Splits 9 Type grouped Missingness 80.3% Domain education Source Other Year 2023
al
14.0 ± 1.0 13.0 ± 0.0 13.0 ± 0.0 13.0 ± 0.0 13.0 ± 0.0 13.0 ± 0.0 13.0 ± 0.0 29.0 ± 46.0 -
17500
tim
17.0 ± 4.0 13.0 ± 0.0 13.0 ± 0.0 13.0 ± 0.0 13.0 ± 0.0 13.0 ± 0.0 13.0 ± 0.0 13415.0 ± 31489.0 13.0 ± 0.0 13.0 ± 0.0 13.0 ± 0.0
Dataset: early_learning_predictors
Tuned + Ens.
0
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Tuned
Test Error (RMSE)
Default
102
103
Train time (s)
104
Figure G.40: early_learning_predictors: per-method test error (left) and HPO Pareto trajectory (right).
0.050 ± 0.017 0.029 ± 0.014 0.029 ± 0.014 0.028 ± 0.013 0.038 ± 0.016 0.036 ± 0.016 0.033 ± 0.015 0.038 ± 0.017 -
Dataset: early_stage_diabetes_risk_predLightGBM
TabM XGBoost TabDPT RealMLP TabPFN-2.6 RandomForest CatBoost ExtraTrees TabICLv2 Metadata Samples 251 Features 16 Problem binary Splits 60 Type IID Missingness 0.0% Domain medical & healthcare Source UCI Year 2019
0.040 0.038 0.036 0.034 0.032 0.030 0.028 0.026 0.024
al
Tuned + Ens.
0.050 ± 0.017 0.029 ± 0.014 0.029 ± 0.015 0.028 ± 0.013 0.038 ± 0.017 0.037 ± 0.016 0.032 ± 0.015 0.037 ± 0.018 -
tim
Tuned
0.053 ± 0.018 0.028 ± 0.014 0.027 ± 0.014 0.028 ± 0.014 0.041 ± 0.017 0.041 ± 0.016 0.037 ± 0.017 0.040 ± 0.017 0.033 ± 0.015 0.031 ± 0.013 0.025 ± 0.012
Test Error (1-AUC)
Default
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
102
Train time (s)
103
Figure G.41: early_stage_diabetes_risk_prediction: per-method test error (left) and HPO Pareto trajectory (right).
Tuned
Tuned + Ens.
0.386 ± 0.057 0.396 ± 0.054 0.420 ± 0.038 0.385 ± 0.060 0.408 ± 0.059 0.443 ± 0.059 0.405 ± 0.097 0.386 ± 0.059 0.372 ± 0.065 0.393 ± 0.077 0.362 ± 0.076
0.388 ± 0.063 0.404 ± 0.064 0.394 ± 0.055 0.369 ± 0.056 0.384 ± 0.058 0.405 ± 0.053 0.382 ± 0.071 0.395 ± 0.062 -
0.384 ± 0.062 0.401 ± 0.061 0.392 ± 0.055 0.368 ± 0.055 0.382 ± 0.054 0.406 ± 0.054 0.371 ± 0.067 0.391 ± 0.063 -
Dataset: ecoli_proteins 0.44
Test Error (logloss)
Default
0.43 0.42 0.41 0.40 0.39 0.38 0.37 0.36
Op tim al
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
102
Train time (s)
XGBoost RandomForest TabPFN-2.6 ExtraTrees TabM LightGBM TabDPT RealMLP CatBoost TabICLv2 Metadata Samples 327 Features 6 Problem multiclass Classes 5 Splits 60 Type IID Missingness 0.0% Domain biology & life sciences Source UCI Year 1996
103
Figure G.42: ecoli_proteins: per-method test error (left) and HPO Pareto trajectory (right). 67
0.295 ± 0.010 0.256 ± 0.007 0.258 ± 0.008 0.253 ± 0.006 0.255 ± 0.006 0.255 ± 0.005 0.258 ± 0.006 0.254 ± 0.008 0.259 ± 0.007 0.254 ± 0.008 0.253 ± 0.008
0.278 ± 0.007 0.253 ± 0.008 0.256 ± 0.009 0.255 ± 0.007 0.256 ± 0.004 0.255 ± 0.004 0.256 ± 0.006 0.255 ± 0.008 -
0.276 ± 0.008 0.254 ± 0.006 0.255 ± 0.008 0.255 ± 0.006 0.256 ± 0.005 0.255 ± 0.005 0.253 ± 0.007 0.254 ± 0.006 -
Dataset: ecommerce_shipping 0.258 0.257 0.256 0.255 0.254 0.253
TabDPT LightGBM XGBoost ExtraTrees TabPFN-2.6 TabM RandomForest RealMLP CatBoost TabICLv2 Metadata Samples 10,999 Features 10 Problem binary Splits 9 Type IID Missingness 0.0% Domain business & marketing Source Kaggle Year 2021
tim al
Tuned + Ens.
Test Error (1-AUC)
Tuned
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
101
102
Train time (s)
103
Figure G.43: ecommerce_shipping: per-method test error (left) and HPO Pareto trajectory (right).
Tuned + Ens.
2.221 2.920 2.460 2.548 2.428 2.520 1.702 2.059 -
2.208 2.947 2.475 2.480 2.358 2.523 1.772 2.039 -
Dataset: electric_motor_temperature_preRandomForest 3.4
XGBoost CatBoost ExtraTrees TabICLv2 LightGBM TabM RealMLP
3.2 3.0 2.8
Metadata Samples 1,296,316 Features 109 Problem regression Splits 1 Type grouped Missingness 0.0% Domain industry & manufacturing Source Kaggle Year 2021
2.6 2.4 2.2
1.8
al
2.0 tim
Test Error (RMSE)
Linear 3.053 RandomForest 3.390 ExtraTrees 2.464 CatBoost 2.493 LightGBM 2.694 XGBoost 2.835 RealMLP 1.858 TabM 2.151 TabDPT TabPFN-2.6 TabICLv2 2.454
Tuned
Op
Default
104
103
105
Train time (s)
Figure G.44: electric_motor_temperature_prediction: per-method test error (left) and HPO Pareto trajectory (right).
Tuned + Ens.
0.036 ± 0.004 0.013 ± 0.001 0.013 ± 0.002 0.012 ± 0.002 0.012 ± 0.002 0.011 ± 0.002 0.010 ± 0.003 0.010 ± 0.002 -
0.034 ± 0.003 0.013 ± 0.001 0.011 ± 0.002 0.012 ± 0.002 0.010 ± 0.002 0.011 ± 0.002 0.007 ± 0.002 0.010 ± 0.002 -
Dataset: emscad
0.013
RandomForest CatBoost TabDPT XGBoost ExtraTrees LightGBM TabM RealMLP TabICLv2 TabPFN-2.6 Metadata Samples 17,460 Features 265 Problem binary Splits 9 Type IID Missingness 28.4% Domain business & marketing Source Other Year 2014
0.012 0.011 0.010 0.009 0.008 0.007 0.006 0.005
tim al
Tuned
0.035 ± 0.003 0.013 ± 0.001 0.013 ± 0.001 0.012 ± 0.001 0.011 ± 0.002 0.012 ± 0.002 0.012 ± 0.004 0.012 ± 0.002 0.011 ± 0.002 0.005 ± 0.001 0.006 ± 0.001
Test Error (1-AUC)
Default
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
102
103
Train time (s)
104
Figure G.45: emscad: per-method test error (left) and HPO Pareto trajectory (right).
0.066 ± 0.025 0.107 ± 0.027 0.095 ± 0.020 0.069 ± 0.025 0.069 ± 0.019 0.092 ± 0.023 0.066 ± 0.019 0.075 ± 0.015 -
0.065 ± 0.025 0.107 ± 0.027 0.096 ± 0.019 0.069 ± 0.025 0.070 ± 0.019 0.092 ± 0.024 0.065 ± 0.019 0.075 ± 0.014 -
ExtraTrees XGBoost TabDPT TabM LightGBM CatBoost TabPFN-2.6 RealMLP TabICLv2 Metadata Samples 366 Features 34 Problem multiclass Classes 6 Splits 60 Type IID Missingness 0.1% Domain medical & healthcare Source UCI Year 1997
0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1
al
0.109 ± 0.016 0.158 ± 0.014 0.172 ± 0.013 0.084 ± 0.031 0.849 ± 0.066 0.121 ± 0.034 0.071 ± 0.018 0.074 ± 0.014 0.083 ± 0.021 0.066 ± 0.029 0.054 ± 0.020
Dataset: eryhemato_squamous_diseaseRandomForest Test Error (logloss)
Tuned + Ens.
tim
Tuned
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
102
103
Train time (s)
Figure G.46: eryhemato_squamous_disease: per-method test error (left) and HPO Pareto trajectory (right). 68
763.9 ± 24.5 733.0 ± 26.9 735.9 ± 29.7 733.7 ± 25.3 740.9 ± 25.0 736.8 ± 26.1 730.6 ± 29.1 753.0 ± 28.1 -
764.1 ± 24.5 731.5 ± 26.9 734.3 ± 28.7 730.2 ± 26.2 731.2 ± 26.3 733.6 ± 26.6 723.5 ± 28.3 748.1 ± 29.1 -
TabM ExtraTrees XGBoost RandomForest LightGBM CatBoost TabPFN-2.6 RealMLP TabDPT TabICLv2 Metadata Samples 1,538 Features 7 Problem regression Splits 30 Type IID Missingness 0.0% Domain technology & internet Source Kaggle Year 2020
750 740 730 720 al
791.7 ± 27.5 745.0 ± 28.0 738.3 ± 29.6 737.3 ± 27.0 742.5 ± 26.7 751.9 ± 29.7 755.3 ± 27.0 750.9 ± 28.6 720.1 ± 30.1 726.8 ± 27.9 714.3 ± 30.7
Dataset: fiat_500 Test Error (RMSE)
Tuned + Ens.
tim
Tuned
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
101
102
Train time (s)
103
Figure G.47: fiat_500: per-method test error (left) and HPO Pareto trajectory (right).
0.181 ± 0.018 0.206 ± 0.018 0.185 ± 0.019 0.187 ± 0.018 0.183 ± 0.019 0.191 ± 0.018 0.184 ± 0.018 0.183 ± 0.018 -
0.181 ± 0.018 0.205 ± 0.018 0.187 ± 0.018 0.187 ± 0.018 0.183 ± 0.019 0.192 ± 0.018 0.182 ± 0.017 0.182 ± 0.018 -
RandomForest XGBoost ExtraTrees CatBoost LightGBM RealMLP TabM TabDPT TabICLv2 TabPFN-2.6 Metadata Samples 1,500 Features 6 Problem binary Splits 30 Type IID Missingness 0.2% Domain business & marketing Source Kaggle Year 2023
0.22 0.21 0.20 0.19 0.18
al
0.181 ± 0.018 0.219 ± 0.018 0.224 ± 0.016 0.185 ± 0.018 0.206 ± 0.018 0.202 ± 0.016 0.188 ± 0.017 0.183 ± 0.018 0.180 ± 0.018 0.179 ± 0.018 0.180 ± 0.017
Dataset: fitness_club
Tuned + Ens.
Test Error (1-AUC)
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Tuned
Op tim
Default
101
102
Train time (s)
103
Figure G.48: fitness_club: per-method test error (left) and HPO Pareto trajectory (right).
8.333 ± 0.068 7.586 ± 0.049 7.778 ± 0.058 7.366 ± 0.055 7.513 ± 0.056 7.392 ± 0.057 7.452 ± 0.057 7.675 ± 0.054 -
8.295 ± 0.066 7.589 ± 0.047 7.762 ± 0.057 7.366 ± 0.054 7.461 ± 0.054 7.396 ± 0.056 7.448 ± 0.056 7.675 ± 0.054 -
ExtraTrees TabM TabICLv2 RandomForest TabDPT LightGBM RealMLP XGBoost TabPFN-2.6 CatBoost Metadata Samples 45,451 Features 9 Problem regression Splits 9 Type IID Missingness 0.0% Domain business & marketing Source Kaggle Year 2023
8.0 7.9 7.8 7.7 7.6 7.5 7.4
al
8.557 ± 0.046 7.774 ± 0.042 8.108 ± 0.044 7.376 ± 0.055 7.618 ± 0.056 7.392 ± 0.057 7.925 ± 0.062 7.784 ± 0.045 7.461 ± 0.070 7.381 ± 0.057 7.593 ± 0.053
Dataset: food_delivery_time 8.1
Test Error (RMSE)
Tuned + Ens.
tim
Tuned
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
101
102
103
Train time (s)
104
Figure G.49: food_delivery_time: per-method test error (left) and HPO Pareto trajectory (right).
0.988 ± 0.119 0.662 ± 0.071 0.639 ± 0.079 0.626 ± 0.089 0.676 ± 0.085 0.732 ± 0.082 0.645 ± 0.105 0.675 ± 0.096 -
0.888 ± 0.101 0.655 ± 0.073 0.635 ± 0.072 0.623 ± 0.084 0.677 ± 0.084 0.726 ± 0.083 0.631 ± 0.084 0.662 ± 0.080 -
LightGBM TabM RandomForest ExtraTrees RealMLP CatBoost TabDPT TabPFN-2.6 TabICLv2 Metadata Samples 214 Features 9 Problem multiclass Classes 6 Splits 60 Type IID Missingness 0.0% Domain chemistry & material science Source UCI Year 1987
0.75 0.70 0.65 0.60 0.55
al
0.964 ± 0.129 0.669 ± 0.063 0.690 ± 0.055 0.636 ± 0.083 0.725 ± 0.070 0.746 ± 0.092 0.788 ± 0.124 0.680 ± 0.091 0.591 ± 0.088 0.560 ± 0.098 0.552 ± 0.093
Dataset: forensic_glass_identificationXGBoost 0.80 Test Error (logloss)
Tuned + Ens.
tim
Tuned
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
102
Train time (s)
103
Figure G.50: forensic_glass_identification: per-method test error (left) and HPO Pareto trajectory (right). 69
1.404 ± 0.079 1.441 ± 0.073 1.407 ± 0.080 1.399 ± 0.080 1.401 ± 0.080 1.399 ± 0.080 1.400 ± 0.079 1.400 ± 0.081 -
RandomForest ExtraTrees TabPFN-2.6 RealMLP LightGBM TabICLv2 TabM XGBoost CatBoost TabDPT
1.45 1.44 1.43
Metadata Samples 517 Features 12 Problem regression Splits 60 Type IID Missingness 0.0% Domain environmental science & climate Source UCI Year 2008
1.42 1.41 1.40 al
1.415 ± 0.088 1.440 ± 0.074 1.405 ± 0.080 1.400 ± 0.079 1.402 ± 0.080 1.399 ± 0.080 1.402 ± 0.080 1.400 ± 0.081 -
1.46
Test Error (RMSE)
1.428 ± 0.075 1.463 ± 0.070 1.444 ± 0.072 1.397 ± 0.080 1.399 ± 0.080 1.398 ± 0.079 1.404 ± 0.078 1.399 ± 0.080 1.392 ± 0.079 1.403 ± 0.082 1.399 ± 0.079
Dataset: forest_fires
Tuned + Ens.
1.39
tim
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Tuned
Op
Default
102
Train time (s)
103
Figure G.51: forest_fires: per-method test error (left) and HPO Pareto trajectory (right).
0.130 ± 0.027 0.153 ± 0.031 0.147 ± 0.032 0.142 ± 0.030 0.136 ± 0.028 0.147 ± 0.030 0.113 ± 0.027 0.135 ± 0.027 -
RandomForest XGBoost ExtraTrees CatBoost LightGBM TabM RealMLP TabDPT TabPFN-2.6 TabICLv2 Metadata Samples 319 Features 38 Problem binary Splits 60 Type IID Missingness 0.0% Domain medical & healthcare Source UCI Year 2023
0.15 0.14 0.13 0.12 0.11 0.10
al
0.146 ± 0.030 0.154 ± 0.032 0.147 ± 0.031 0.142 ± 0.029 0.140 ± 0.028 0.149 ± 0.031 0.125 ± 0.027 0.134 ± 0.027 -
0.16
Test Error (1-AUC)
0.142 ± 0.032 0.152 ± 0.032 0.162 ± 0.041 0.147 ± 0.030 0.152 ± 0.031 0.158 ± 0.030 0.122 ± 0.025 0.135 ± 0.028 0.107 ± 0.026 0.104 ± 0.026 0.100 ± 0.029
Dataset: gallstone_disease
Tuned + Ens.
tim
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Tuned
Op
Default
102
Train time (s)
103
Figure G.52: gallstone_disease: per-method test error (left) and HPO Pareto trajectory (right).
Tuned
Tuned + Ens.
0.27 ± 0.66 0.13 ± 0.03 0.13 ± 0.03 0.13 ± 0.03 0.13 ± 0.03 0.13 ± 0.03 0.14 ± 0.03 73.83 ± 284.20 0.13 ± 0.03 0.13 ± 0.03 0.13 ± 0.04
0.19 ± 0.12 0.13 ± 0.03 0.13 ± 0.03 0.13 ± 0.03 0.13 ± 0.03 0.13 ± 0.03 0.13 ± 0.04 0.22 ± 0.38 -
0.22 ± 0.35 0.13 ± 0.03 0.13 ± 0.03 0.13 ± 0.03 0.13 ± 0.03 0.13 ± 0.03 0.13 ± 0.03 0.13 ± 0.04 -
Dataset: garments_worker_productivityRealMLP
TabICLv2 TabM ExtraTrees TabDPT RandomForest CatBoost LightGBM XGBoost TabPFN-2.6 Metadata Samples 1,197 Features 24 Problem regression Splits 30 Type temporal Missingness 3.6% Domain industry & manufacturing Source UCI Year 2020
Test Error (RMSE)
70 60 50 40 30 20 tim al
10 0
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
101
102
Train time (s)
103
Figure G.53: garments_worker_productivity: per-method test error (left) and HPO Pareto trajectory (right).
0.206 ± 0.114 0.242 ± 0.136 0.236 ± 0.141 0.189 ± 0.106 0.182 ± 0.102 0.214 ± 0.131 0.174 ± 0.092 0.186 ± 0.107 -
Dataset: ghanas_indigenous_intelTabDPT
TabICLv2 RandomForest ExtraTrees XGBoost TabPFN-2.6 CatBoost TabM LightGBM RealMLP
0.24 0.23 0.22 0.21
Metadata Samples 10,928 Features 19 Problem multiclass Classes 4 Splits 6 Type temporal Missingness 29.2% Domain environmental science & climate Source Zindi Year 2025
0.20 0.19 0.18 al
Tuned + Ens.
0.209 ± 0.110 0.251 ± 0.144 0.232 ± 0.134 0.184 ± 0.108 0.191 ± 0.102 0.216 ± 0.136 0.174 ± 0.091 0.202 ± 0.126 -
Test Error (logloss)
Tuned
0.231 ± 0.138 0.239 ± 0.125 0.218 ± 0.108 0.204 ± 0.123 0.225 ± 0.158 0.236 ± 0.119 0.171 ± 0.118 0.183 ± 0.106 0.244 ± 0.091 0.193 ± 0.092 0.243 ± 0.082
0.17
tim
Default
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
101
102
Train time (s)
103
Figure G.54: ghanas_indigenous_intel: per-method test error (left) and HPO Pareto trajectory (right). 70
0.175 ± 0.003 0.178 ± 0.003 0.181 ± 0.004 0.156 ± 0.004 0.153 ± 0.004 0.154 ± 0.004 0.154 ± 0.004 0.153 ± 0.003 -
0.173 ± 0.003 0.177 ± 0.002 0.181 ± 0.004 0.156 ± 0.003 0.152 ± 0.004 0.154 ± 0.003 0.154 ± 0.003 0.153 ± 0.004 -
0.22 0.21 0.20 0.19 0.18 0.17 0.16
TabDPT ExtraTrees RandomForest TabPFN-2.6 CatBoost XGBoost RealMLP TabICLv2 TabM LightGBM Metadata Samples 150,000 Features 10 Problem binary Splits 9 Type IID Missingness 2.2% Domain finance Source Kaggle Year 2011
al
0.176 ± 0.003 0.225 ± 0.020 0.212 ± 0.011 0.157 ± 0.003 0.153 ± 0.004 0.154 ± 0.004 0.156 ± 0.003 0.154 ± 0.004 0.222 ± 0.010 0.169 ± 0.006 0.153 ± 0.003
Dataset: give_me_some_credit
Tuned + Ens.
Test Error (1-AUC)
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Tuned
0.15
Op tim
Default
101
102
104
103
Train time (s)
Figure G.55: give_me_some_credit: per-method test error (left) and HPO Pareto trajectory (right).
0.047 ± 0.006 0.042 ± 0.006 0.037 ± 0.006 0.026 ± 0.006 0.023 ± 0.006 0.026 ± 0.006 0.016 ± 0.004 0.018 ± 0.005 -
0.047 ± 0.006 0.041 ± 0.006 0.037 ± 0.006 0.027 ± 0.006 0.023 ± 0.006 0.026 ± 0.006 0.015 ± 0.004 0.018 ± 0.005 -
ExtraTrees CatBoost XGBoost LightGBM TabM RealMLP TabPFN-2.6 TabDPT TabICLv2 Metadata Samples 2,400 Features 30 Problem binary Splits 30 Type IID Missingness 0.0% Domain biology & life sciences Source OpenML Year 2020
0.04 0.03 0.02 0.01 al
0.051 ± 0.007 0.043 ± 0.007 0.047 ± 0.007 0.027 ± 0.006 0.028 ± 0.005 0.028 ± 0.006 0.016 ± 0.005 0.033 ± 0.006 0.007 ± 0.003 0.008 ± 0.003 0.005 ± 0.002
Dataset: hazelnut_spread_contaminant_deRandomForest
tim
Tuned + Ens.
Op
Tuned
Test Error (1-AUC)
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
101
102
103
Train time (s)
Figure G.56: hazelnut_spread_contaminant_detection: per-method test error (left) and HPO Pareto trajectory (right).
Tuned + Ens.
0.446 ± 0.024 0.379 ± 0.028 0.372 ± 0.031 0.372 ± 0.030 0.370 ± 0.030 0.370 ± 0.030 0.374 ± 0.029 0.371 ± 0.031 -
0.446 ± 0.024 0.378 ± 0.028 0.372 ± 0.031 0.372 ± 0.030 0.368 ± 0.031 0.371 ± 0.030 0.371 ± 0.030 0.371 ± 0.031 -
Dataset: healthcare_insurance_expenses RandomForest 0.395
CatBoost ExtraTrees TabM XGBoost RealMLP LightGBM TabDPT TabPFN-2.6 TabICLv2 Metadata Samples 1,338 Features 6 Problem regression Splits 30 Type IID Missingness 0.0% Domain medical & healthcare Source Kaggle Year 2023
0.390 0.385 0.380 0.375 0.370 0.365 0.360
al
Tuned
0.446 ± 0.024 0.394 ± 0.026 0.390 ± 0.027 0.373 ± 0.031 0.377 ± 0.028 0.384 ± 0.026 0.372 ± 0.031 0.373 ± 0.030 0.364 ± 0.032 0.360 ± 0.037 0.360 ± 0.034
Test Error (RMSE)
Default
Op tim
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
101
102
103
Train time (s)
Figure G.57: healthcare_insurance_expenses: per-method test error (left) and HPO Pareto trajectory (right).
0.093 ± 0.025 0.100 ± 0.028 0.092 ± 0.026 0.094 ± 0.027 0.094 ± 0.026 0.101 ± 0.028 0.093 ± 0.026 0.095 ± 0.026 -
0.090 ± 0.025 0.100 ± 0.027 0.092 ± 0.026 0.094 ± 0.026 0.092 ± 0.026 0.100 ± 0.027 0.092 ± 0.026 0.094 ± 0.027 -
RandomForest CatBoost TabDPT TabM RealMLP LightGBM TabICLv2 TabPFN-2.6 ExtraTrees Metadata Samples 303 Features 13 Problem binary Splits 60 Type IID Missingness 0.2% Domain medical & healthcare Source UCI Year 1989
0.110 0.105 0.100 0.095 0.090
al
0.092 ± 0.025 0.097 ± 0.027 0.090 ± 0.025 0.098 ± 0.028 0.113 ± 0.028 0.104 ± 0.028 0.097 ± 0.025 0.093 ± 0.026 0.093 ± 0.026 0.090 ± 0.027 0.091 ± 0.026
Dataset: heart_disease_cleveland XGBoost
tim
Tuned + Ens.
Test Error (1-AUC)
Tuned
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
102
Train time (s)
103
Figure G.58: heart_disease_cleveland: per-method test error (left) and HPO Pareto trajectory (right). 71
0.093 ± 0.026 0.110 ± 0.024 0.096 ± 0.023 0.099 ± 0.024 0.099 ± 0.024 0.101 ± 0.022 0.091 ± 0.023 0.095 ± 0.022 -
0.14 0.13 0.12 0.11 0.10 0.09
RandomForest XGBoost LightGBM CatBoost ExtraTrees TabDPT TabICLv2 TabM TabPFN-2.6 RealMLP Metadata Samples 294 Features 13 Problem binary Splits 60 Type IID Missingness 20.5% Domain medical & healthcare Source UCI Year 1989
al
0.096 ± 0.028 0.109 ± 0.025 0.096 ± 0.023 0.100 ± 0.025 0.099 ± 0.026 0.101 ± 0.022 0.091 ± 0.023 0.093 ± 0.023 -
0.15
Test Error (1-AUC)
0.097 ± 0.026 0.110 ± 0.023 0.102 ± 0.022 0.098 ± 0.021 0.154 ± 0.032 0.124 ± 0.025 0.094 ± 0.023 0.094 ± 0.023 0.095 ± 0.021 0.093 ± 0.024 0.094 ± 0.023
Dataset: heart_disease_hungary
Tuned + Ens.
tim
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Tuned
Op
Default
102
Train time (s)
103
Figure G.59: heart_disease_hungary: per-method test error (left) and HPO Pareto trajectory (right).
0.334 ± 0.063 0.327 ± 0.051 0.311 ± 0.049 0.321 ± 0.048 0.333 ± 0.065 0.358 ± 0.063 0.348 ± 0.060 0.347 ± 0.065 -
0.335 ± 0.061 0.327 ± 0.050 0.311 ± 0.050 0.318 ± 0.048 0.334 ± 0.064 0.349 ± 0.062 0.351 ± 0.055 0.344 ± 0.061 -
RealMLP LightGBM TabICLv2 TabM RandomForest TabPFN-2.6 CatBoost ExtraTrees TabDPT Metadata Samples 200 Features 13 Problem binary Splits 60 Type IID Missingness 26.8% Domain medical & healthcare Source UCI Year 1989
0.36 0.35 0.34 0.33 0.32 0.31 0.30
al
0.338 ± 0.056 0.322 ± 0.049 0.304 ± 0.047 0.305 ± 0.050 0.371 ± 0.064 0.368 ± 0.061 0.342 ± 0.057 0.324 ± 0.057 0.294 ± 0.049 0.319 ± 0.055 0.327 ± 0.051
Dataset: heart_disease_va_long_beachXGBoost
0.37
Test Error (1-AUC)
Tuned + Ens.
tim
Tuned
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
102
Train time (s)
103
Figure G.60: heart_disease_va_long_beach: per-method test error (left) and HPO Pareto trajectory (right).
Tuned + Ens.
0.129 ± 0.028 0.087 ± 0.023 0.092 ± 0.023 0.092 ± 0.022 0.092 ± 0.023 0.090 ± 0.021 0.099 ± 0.024 0.090 ± 0.024 -
0.128 ± 0.027 0.087 ± 0.022 0.091 ± 0.022 0.090 ± 0.022 0.091 ± 0.022 0.089 ± 0.021 0.095 ± 0.020 0.089 ± 0.024 -
Dataset: heart_failure_followup_survivalRealMLP
TabDPT ExtraTrees CatBoost LightGBM TabPFN-2.6 XGBoost TabM RandomForest TabICLv2 Metadata Samples 299 Features 12 Problem binary Splits 60 Type IID Missingness 0.0% Domain medical & healthcare Source UCI Year 2020
0.120 0.115 0.110 0.105 0.100 0.095 0.090 0.085
al
Tuned
0.129 ± 0.029 0.086 ± 0.021 0.106 ± 0.021 0.090 ± 0.021 0.090 ± 0.021 0.096 ± 0.025 0.121 ± 0.026 0.088 ± 0.023 0.094 ± 0.022 0.089 ± 0.020 0.084 ± 0.020
Test Error (1-AUC)
Default
Op tim
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
102
Train time (s)
103
Figure G.61: heart_failure_followup_survival: per-method test error (left) and HPO Pareto trajectory (right).
0.214 ± 0.005 0.207 ± 0.004 0.206 ± 0.005 0.201 ± 0.005 0.201 ± 0.005 0.202 ± 0.004 0.199 ± 0.004 0.201 ± 0.004 -
Dataset: heloc
RandomForest ExtraTrees TabDPT XGBoost TabM CatBoost LightGBM RealMLP TabPFN-2.6 TabICLv2 Metadata Samples 10,459 Features 23 Problem binary Splits 9 Type IID Missingness 0.0% Domain finance Source Kaggle Year 2021
0.206 0.204 0.202 0.200 0.198
al
Tuned + Ens.
0.214 ± 0.006 0.207 ± 0.004 0.207 ± 0.004 0.201 ± 0.005 0.201 ± 0.005 0.203 ± 0.004 0.200 ± 0.005 0.202 ± 0.004 -
tim
Tuned
0.214 ± 0.005 0.207 ± 0.004 0.207 ± 0.005 0.202 ± 0.005 0.205 ± 0.004 0.206 ± 0.004 0.202 ± 0.005 0.203 ± 0.004 0.204 ± 0.005 0.198 ± 0.004 0.198 ± 0.004
Test Error (1-AUC)
Default
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
101
102
Train time (s)
103
Figure G.62: heloc: per-method test error (left) and HPO Pareto trajectory (right). 72
0.186 ± 0.034 0.154 ± 0.019 0.147 ± 0.013 0.149 ± 0.022 0.144 ± 0.022 0.156 ± 0.026 0.141 ± 0.026 0.134 ± 0.026 -
TabDPT XGBoost RandomForest CatBoost ExtraTrees LightGBM RealMLP TabM TabICLv2 TabPFN-2.6 Metadata Samples 608 Features 12 Problem multiclass Classes 4 Splits 60 Type IID Missingness 0.4% Domain medical & healthcare Source UCI Year 2018
0.17 0.16 0.15 0.14 0.13
al
0.196 ± 0.037 0.155 ± 0.023 0.147 ± 0.013 0.150 ± 0.024 0.144 ± 0.023 0.157 ± 0.027 0.144 ± 0.029 0.136 ± 0.027 -
0.18
Test Error (logloss)
0.195 ± 0.044 0.162 ± 0.013 0.184 ± 0.010 0.150 ± 0.022 0.160 ± 0.025 0.164 ± 0.028 0.170 ± 0.039 0.150 ± 0.026 0.160 ± 0.030 0.125 ± 0.026 0.131 ± 0.034
Dataset: hepatitis_c_prediction
Tuned + Ens.
tim
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Tuned
Op
Default
102
Train time (s)
103
Figure G.63: hepatitis_c_prediction: per-method test error (left) and HPO Pareto trajectory (right).
0.143 ± 0.062 0.141 ± 0.055 0.138 ± 0.048 0.132 ± 0.050 0.150 ± 0.056 0.140 ± 0.058 0.150 ± 0.066 0.136 ± 0.060 -
0.146 ± 0.059 0.142 ± 0.054 0.135 ± 0.046 0.132 ± 0.050 0.150 ± 0.055 0.143 ± 0.059 0.145 ± 0.064 0.131 ± 0.057 -
XGBoost RealMLP TabPFN-2.6 CatBoost TabM RandomForest ExtraTrees TabICLv2 TabDPT Metadata Samples 155 Features 19 Problem binary Splits 60 Type IID Missingness 5.7% Domain medical & healthcare Source UCI Year 1981
0.16 0.15 0.14 0.13 al
0.156 ± 0.058 0.130 ± 0.048 0.128 ± 0.044 0.137 ± 0.049 0.170 ± 0.057 0.168 ± 0.061 0.146 ± 0.060 0.131 ± 0.057 0.121 ± 0.045 0.135 ± 0.054 0.127 ± 0.055
Dataset: hepatitis_survival_prediction LightGBM
0.17
Test Error (1-AUC)
Tuned + Ens.
0.12
tim
Tuned
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
102
Train time (s)
103
Figure G.64: hepatitis_survival_prediction: per-method test error (left) and HPO Pareto trajectory (right).
0.266 ± 0.031 0.213 ± 0.027 0.214 ± 0.022 0.214 ± 0.021 0.209 ± 0.034 0.209 ± 0.028 0.220 ± 0.030 0.232 ± 0.029 -
Dataset: hiva_agnostic
0.25
TabICLv2 TabM RealMLP TabDPT TabPFN-2.6 RandomForest CatBoost ExtraTrees XGBoost LightGBM Metadata Samples 3,845 Features 1,518 Problem binary Splits 9 Type IID Missingness 0.0% Domain chemistry & material science Source Other Year 2007
0.24 0.23 0.22 0.21
al
Tuned + Ens.
0.266 ± 0.031 0.212 ± 0.024 0.215 ± 0.021 0.215 ± 0.021 0.210 ± 0.038 0.213 ± 0.029 0.240 ± 0.043 0.232 ± 0.027 -
Test Error (1-AUC)
Tuned
0.262 ± 0.026 0.212 ± 0.024 0.208 ± 0.025 0.210 ± 0.023 0.208 ± 0.033 0.208 ± 0.031 0.233 ± 0.030 0.237 ± 0.032 0.214 ± 0.025 0.212 ± 0.020 0.250 ± 0.028
tim
Default
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
101
102
104
103
Train time (s)
Figure G.65: hiva_agnostic: per-method test error (left) and HPO Pareto trajectory (right).
0.226 ± 0.001 0.260 ± 0.002 0.257 ± 0.002 0.209 ± 0.001 0.213 ± 0.001 0.212 ± 0.001 0.216 ± 0.001 0.211 ± 0.001 0.224 ± 0.002
0.225 ± 0.001 0.252 ± 0.002 0.252 ± 0.002 0.208 ± 0.001 0.208 ± 0.001 0.208 ± 0.001 0.209 ± 0.001 0.209 ± 0.001 -
0.222 ± 0.001 0.245 ± 0.002 0.244 ± 0.002 0.208 ± 0.001 0.208 ± 0.001 0.208 ± 0.001 0.208 ± 0.001 0.208 ± 0.001 -
Dataset: home_credit_default_risk
0.26 0.25 0.24 0.23 0.22 0.21
RandomForest ExtraTrees TabICLv2 TabM CatBoost LightGBM RealMLP XGBoost Metadata Samples 307,507 Features 504 Problem binary Splits 9 Type IID Missingness 33.7% Domain finance Source Kaggle Year 2018
tim al
Tuned + Ens.
Test Error (1-AUC)
Tuned
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
103
104
Train time (s)
105
Figure G.66: home_credit_default_risk: per-method test error (left) and HPO Pareto trajectory (right). 73
Default
Tuned
Dataset: home_credit_default_stability_
Tuned + Ens.
0.18
0.153 0.169 0.169 0.131 0.136 0.128 0.128 0.130 -
Test Error (1-AUC)
0.17 0.16 0.15 0.14 0.13
RandomForest ExtraTrees TabICLv2 LightGBM CatBoost TabM RealMLP XGBoost Metadata Samples 1,224,927 Features 720 Problem binary Splits 1 Type temporal Missingness 32.8% Domain finance Source Kaggle Year 2024
tim al
0.157 0.172 0.181 0.130 0.138 0.129 0.127 0.131 -
Op
Linear 0.160 RandomForest 0.172 ExtraTrees 0.181 CatBoost 0.134 LightGBM 0.146 XGBoost 0.133 RealMLP 0.145 TabM 0.130 TabDPT TabPFN-2.6 TabICLv2 0.164
104
103
105
Train time (s)
Figure G.67: home_credit_default_stability_1m: per-method test error (left) and HPO Pareto trajectory (right).
Tuned + Ens.
0.200 ± 0.010 0.042 ± 0.005 0.034 ± 0.004 0.030 ± 0.004 0.032 ± 0.002 0.032 ± 0.003 0.013 ± 0.003 0.026 ± 0.004 -
0.197 ± 0.009 0.042 ± 0.005 0.034 ± 0.004 0.029 ± 0.004 0.032 ± 0.003 0.032 ± 0.004 0.010 ± 0.002 0.025 ± 0.003 -
Dataset: homeq_default_prediction RandomForest
ExtraTrees XGBoost LightGBM CatBoost TabM RealMLP TabDPT TabICLv2 TabPFN-2.6 Metadata Samples 5,708 Features 12 Problem binary Splits 9 Type IID Missingness 6.1% Domain business & marketing Source Other Year 2016
0.040 0.035 0.030 0.025 0.020 0.015 0.010 0.005 0.000
tim al
Tuned
0.206 ± 0.010 0.042 ± 0.005 0.034 ± 0.004 0.032 ± 0.003 0.035 ± 0.004 0.039 ± 0.004 0.023 ± 0.005 0.033 ± 0.004 0.007 ± 0.002 0.001 ± 0.000 0.003 ± 0.001
Test Error (1-AUC)
Default
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
101
102
103
Train time (s)
Figure G.68: homeq_default_prediction: per-method test error (left) and HPO Pareto trajectory (right).
Tuned + Ens.
0.050 ± 0.000 0.040 ± 0.001 0.039 ± 0.001 0.032 ± 0.001 0.033 ± 0.001 0.033 ± 0.001 0.033 ± 0.001 0.033 ± 0.001 -
0.050 ± 0.000 0.039 ± 0.000 0.039 ± 0.001 0.032 ± 0.001 0.033 ± 0.001 0.033 ± 0.001 0.033 ± 0.001 0.033 ± 0.001 -
Dataset: homesite_quote_conversion 0.044 0.042 0.040 0.038 0.036 0.034 0.032
TabICLv2 RandomForest ExtraTrees LightGBM XGBoost TabM RealMLP CatBoost Metadata Samples 260,753 Features 304 Problem binary Splits 9 Type IID Missingness 4.6% Domain insurance Source Kaggle Year 2015
al
Tuned
0.056 ± 0.000 0.043 ± 0.001 0.045 ± 0.001 0.032 ± 0.000 0.034 ± 0.001 0.033 ± 0.001 0.034 ± 0.001 0.033 ± 0.001 0.042 ± 0.001
Test Error (1-AUC)
Default
Op tim
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
102
103
104
Train time (s)
105
Figure G.69: homesite_quote_conversion: per-method test error (left) and HPO Pareto trajectory (right).
0.749 ± 0.044 0.722 ± 0.036 0.725 ± 0.041 0.733 ± 0.032 0.712 ± 0.037 0.713 ± 0.038 0.707 ± 0.037 0.696 ± 0.036 -
Dataset: horse_colic_survival
0.82 0.80 0.78 0.76 0.74 0.72 0.70
al
Tuned + Ens.
0.750 ± 0.046 0.722 ± 0.034 0.730 ± 0.050 0.735 ± 0.033 0.714 ± 0.039 0.713 ± 0.038 0.717 ± 0.042 0.696 ± 0.037 -
Test Error (logloss)
Tuned
0.854 ± 0.081 0.725 ± 0.035 0.727 ± 0.033 0.732 ± 0.032 0.731 ± 0.033 0.734 ± 0.034 0.820 ± 0.051 0.702 ± 0.033 0.726 ± 0.045 0.720 ± 0.049 0.701 ± 0.049
tim
Default
CatBoost TabDPT ExtraTrees RandomForest TabPFN-2.6 XGBoost LightGBM RealMLP TabICLv2 TabM Metadata Samples 344 Features 20 Problem multiclass Classes 3 Splits 60 Type IID Missingness 26.7% Domain biology & life sciences Source UCI Year 1989
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
102
Train time (s)
103
Figure G.70: horse_colic_survival: per-method test error (left) and HPO Pareto trajectory (right). 74
0.207 ± 0.022 0.273 ± 0.033 0.275 ± 0.030 0.182 ± 0.021 0.199 ± 0.022 0.197 ± 0.025 0.193 ± 0.022 0.193 ± 0.024 -
0.30 0.28 0.26 0.24 0.22 0.20 0.18
TabDPT ExtraTrees RandomForest TabICLv2 TabPFN-2.6 LightGBM XGBoost TabM RealMLP CatBoost Metadata Samples 81,418 Features 49 Problem binary Splits 9 Type temporal Missingness 0.0% Domain business & marketing Source Other Year 2019
al
0.207 ± 0.022 0.269 ± 0.035 0.275 ± 0.029 0.181 ± 0.021 0.201 ± 0.022 0.195 ± 0.025 0.208 ± 0.027 0.200 ± 0.021 -
0.32
Test Error (1-AUC)
0.207 ± 0.020 0.278 ± 0.035 0.304 ± 0.033 0.179 ± 0.017 0.203 ± 0.022 0.198 ± 0.026 0.183 ± 0.023 0.190 ± 0.025 0.324 ± 0.040 0.213 ± 0.026 0.223 ± 0.024
Dataset: hotel_booking_demand
Tuned + Ens.
tim
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Tuned
Op
Default
102
104
103
Train time (s)
Figure G.71: hotel_booking_demand: per-method test error (left) and HPO Pareto trajectory (right).
Tuned + Ens.
0.313 ± 0.005 0.224 ± 0.004 0.231 ± 0.004 0.208 ± 0.004 0.210 ± 0.004 0.212 ± 0.004 0.199 ± 0.004 0.204 ± 0.003 -
0.312 ± 0.005 0.224 ± 0.004 0.231 ± 0.004 0.207 ± 0.004 0.211 ± 0.004 0.211 ± 0.004 0.198 ± 0.004 0.202 ± 0.003 -
Dataset: houses
ExtraTrees RandomForest XGBoost LightGBM CatBoost TabM TabDPT RealMLP TabPFN-2.6 TabICLv2 Metadata Samples 19,675 Features 8 Problem regression Splits 9 Type IID Missingness 0.0% Domain business & marketing Source Other Year 1990
0.23 0.22 0.21 0.20 al
Tuned
0.313 ± 0.005 0.224 ± 0.004 0.235 ± 0.004 0.208 ± 0.004 0.213 ± 0.004 0.211 ± 0.004 0.221 ± 0.005 0.209 ± 0.003 0.202 ± 0.003 0.194 ± 0.004 0.191 ± 0.004
Test Error (RMSE)
Default
0.19
Op tim
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
101
102
104
103
Train time (s)
Figure G.72: houses: per-method test error (left) and HPO Pareto trajectory (right).
0.204 ± 0.004 0.208 ± 0.003 0.212 ± 0.004 0.195 ± 0.003 0.199 ± 0.004 0.195 ± 0.003 0.199 ± 0.003 0.199 ± 0.003 0.197 ± 0.004 0.194 ± 0.004 0.193 ± 0.004
0.203 ± 0.003 0.198 ± 0.002 0.200 ± 0.003 0.196 ± 0.003 0.199 ± 0.004 0.196 ± 0.003 0.197 ± 0.004 0.199 ± 0.003 -
0.203 ± 0.003 0.198 ± 0.002 0.199 ± 0.003 0.196 ± 0.003 0.197 ± 0.003 0.195 ± 0.003 0.197 ± 0.004 0.198 ± 0.003 -
Dataset: hr_analytics
0.2125
ExtraTrees RandomForest TabM LightGBM TabDPT RealMLP CatBoost XGBoost TabPFN-2.6 TabICLv2 Metadata Samples 19,158 Features 12 Problem binary Splits 9 Type IID Missingness 9.0% Domain business & marketing Source Kaggle Year 2021
0.2100 0.2075 0.2050 0.2025 0.2000 0.1975 0.1950 al
Tuned + Ens.
0.1925
Op tim
Tuned
Test Error (1-AUC)
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
101
102
Train time (s)
103
Figure G.73: hr_analytics: per-method test error (left) and HPO Pareto trajectory (right).
0.140 ± 0.009 0.140 ± 0.010 0.123 ± 0.007 0.092 ± 0.010 0.105 ± 0.015 0.091 ± 0.012 0.104 ± 0.014 0.110 ± 0.010 -
0.136 ± 0.009 0.133 ± 0.012 0.121 ± 0.005 0.091 ± 0.006 0.093 ± 0.012 0.091 ± 0.016 0.093 ± 0.010 0.103 ± 0.011 -
0.14 0.13 0.12 0.11 0.10
RandomForest TabICLv2 ExtraTrees TabM LightGBM RealMLP XGBoost CatBoost Metadata Samples 590,540 Features 444 Problem binary Splits 3 Type temporal Missingness 45.0% Domain finance Source Kaggle Year 2019
al
0.141 ± 0.009 0.145 ± 0.038 0.123 ± 0.007 0.096 ± 0.010 0.120 ± 0.017 0.103 ± 0.010 0.110 ± 0.015 0.111 ± 0.010 0.121 ± 0.014
Dataset: ieee_fraud_detection
0.09
tim
Tuned + Ens.
Test Error (1-AUC)
Tuned
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
102
103
104
Train time (s)
105
Figure G.74: ieee_fraud_detection: per-method test error (left) and HPO Pareto trajectory (right). 75
Default
Tuned
Tuned + Ens.
0.473 ± 0.007 0.454 ± 0.007 0.462 ± 0.008 0.352 ± 0.006 0.364 ± 0.006 0.369 ± 0.006 0.367 ± 0.006 0.354 ± 0.007 0.440 ± 0.008 0.346 ± 0.005 0.352 ± 0.005
0.472 ± 0.007 0.434 ± 0.007 0.449 ± 0.008 0.353 ± 0.006 0.358 ± 0.007 0.359 ± 0.007 0.351 ± 0.007 0.353 ± 0.007 -
0.470 ± 0.008 0.436 ± 0.006 0.448 ± 0.008 0.352 ± 0.006 0.356 ± 0.007 0.357 ± 0.007 0.348 ± 0.007 0.350 ± 0.007 -
Dataset: immoscout_german_house_prices ExtraTrees
TabDPT RandomForest XGBoost LightGBM TabICLv2 CatBoost TabM RealMLP TabPFN-2.6 Metadata Samples 10,317 Features 147 Problem regression Splits 9 Type IID Missingness 18.8% Domain business & marketing Source Kaggle Year 2019
0.44 0.42 0.40 0.38 0.36 Op
tim
al
Test Error (RMSE)
0.46
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
102
104
103
Train time (s)
Figure G.75: immoscout_german_house_prices: per-method test error (left) and HPO Pareto trajectory (right).
0.264 ± 0.006 0.182 ± 0.009 0.195 ± 0.008 0.156 ± 0.006 0.160 ± 0.006 0.158 ± 0.006 0.153 ± 0.006 0.149 ± 0.005 -
0.264 ± 0.006 0.177 ± 0.007 0.189 ± 0.008 0.154 ± 0.006 0.157 ± 0.005 0.157 ± 0.005 0.150 ± 0.005 0.148 ± 0.005 -
TabDPT RandomForest XGBoost LightGBM TabICLv2 CatBoost TabPFN-2.6 RealMLP TabM Metadata Samples 12,684 Features 24 Problem binary Splits 9 Type IID Missingness 4.4% Domain business & marketing Source UCI Year 2017
0.20 0.19 0.18 0.17 0.16 0.15
al
0.264 ± 0.006 0.186 ± 0.007 0.202 ± 0.008 0.160 ± 0.005 0.162 ± 0.005 0.166 ± 0.006 0.162 ± 0.004 0.151 ± 0.005 0.188 ± 0.008 0.150 ± 0.006 0.154 ± 0.006
Dataset: in_vehicle_coupon_recommendatiExtraTrees
tim
Tuned + Ens.
Test Error (1-AUC)
Tuned
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
101
102
104
103
Train time (s)
Figure G.76: in_vehicle_coupon_recommendation: per-method test error (left) and HPO Pareto trajectory (right).
Dataset: indian_liver_patient_dataset XGBoost
Tuned
Tuned + Ens.
0.275
0.257 ± 0.030 0.256 ± 0.029 0.241 ± 0.030 0.255 ± 0.032 0.273 ± 0.031 0.273 ± 0.032 0.262 ± 0.031 0.255 ± 0.031 0.236 ± 0.029 0.251 ± 0.031 0.234 ± 0.028
0.263 ± 0.029 0.261 ± 0.029 0.247 ± 0.029 0.252 ± 0.031 0.266 ± 0.032 0.269 ± 0.028 0.259 ± 0.031 0.256 ± 0.031 -
0.255 ± 0.030 0.261 ± 0.029 0.245 ± 0.031 0.251 ± 0.032 0.265 ± 0.030 0.269 ± 0.029 0.253 ± 0.031 0.252 ± 0.031 -
0.270
LightGBM RandomForest RealMLP TabM TabPFN-2.6 CatBoost ExtraTrees TabDPT TabICLv2 Metadata Samples 583 Features 10 Problem binary Splits 60 Type IID Missingness 0.1% Domain medical & healthcare Source UCI Year 2012
0.265 0.260 0.255 0.250 0.245
0.235
al
0.240 tim
Test Error (1-AUC)
Default
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
102
Train time (s)
103
Figure G.77: indian_liver_patient_dataset: per-method test error (left) and HPO Pareto trajectory (right).
0.066 ± 0.007 0.017 ± 0.003 0.014 ± 0.002 0.011 ± 0.002 0.013 ± 0.003 0.014 ± 0.003 0.007 ± 0.002 0.008 ± 0.002 -
Dataset: iranian_churn 0.016 0.014 0.012 0.010 0.008 0.006 0.004
RandomForest XGBoost ExtraTrees LightGBM CatBoost TabM RealMLP TabDPT TabPFN-2.6 TabICLv2 Metadata Samples 2,850 Features 13 Problem binary Splits 30 Type IID Missingness 0.0% Domain business & marketing Source UCI Year 2011
al
Tuned + Ens.
0.067 ± 0.007 0.017 ± 0.003 0.015 ± 0.003 0.011 ± 0.002 0.015 ± 0.003 0.014 ± 0.003 0.007 ± 0.002 0.009 ± 0.003 -
tim
Tuned
0.074 ± 0.008 0.016 ± 0.003 0.015 ± 0.002 0.013 ± 0.002 0.014 ± 0.003 0.016 ± 0.003 0.010 ± 0.003 0.015 ± 0.004 0.005 ± 0.002 0.005 ± 0.002 0.004 ± 0.002
Test Error (1-AUC)
Default
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
101
102
Train time (s)
103
Figure G.78: iranian_churn: per-method test error (left) and HPO Pareto trajectory (right). 76
0.277 ± 0.004 0.251 ± 0.007 0.240 ± 0.005 0.251 ± 0.005 0.253 ± 0.005 0.251 ± 0.006 0.256 ± 0.007 0.266 ± 0.004 -
0.277 ± 0.004 0.242 ± 0.005 0.236 ± 0.006 0.251 ± 0.003 0.251 ± 0.004 0.250 ± 0.005 0.242 ± 0.006 0.258 ± 0.004 -
TabM LightGBM CatBoost XGBoost RealMLP RandomForest ExtraTrees TabPFN-2.6 TabDPT TabICLv2 Metadata Samples 10,885 Features 21 Problem binary Splits 9 Type IID Missingness 0.0% Domain technology & internet Source OpenML Year 2004
0.26 0.25 0.24 0.23 al
0.277 ± 0.004 0.249 ± 0.007 0.243 ± 0.007 0.255 ± 0.003 0.252 ± 0.005 0.251 ± 0.004 0.269 ± 0.004 0.267 ± 0.005 0.224 ± 0.006 0.228 ± 0.004 0.223 ± 0.005
Dataset: jm1
0.27
Test Error (1-AUC)
Tuned + Ens.
tim
Tuned
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
101
102
Train time (s)
103
Figure G.79: jm1: per-method test error (left) and HPO Pareto trajectory (right).
Tuned + Ens.
0.184 ± 0.013 0.193 ± 0.005 0.188 ± 0.011 0.159 ± 0.011 0.179 ± 0.011 0.167 ± 0.012 0.183 ± 0.007 0.185 ± 0.009 -
0.183 ± 0.012 0.193 ± 0.007 0.181 ± 0.010 0.159 ± 0.011 0.175 ± 0.009 0.165 ± 0.010 0.180 ± 0.008 0.183 ± 0.009 -
Dataset: kdd_cup_09_appetency 0.28 0.26 0.24 0.22 0.20 0.18 0.16
TabDPT RandomForest TabM ExtraTrees RealMLP LightGBM TabPFN-2.6 TabICLv2 XGBoost CatBoost Metadata Samples 50,000 Features 212 Problem binary Splits 9 Type IID Missingness 67.2% Domain business & marketing Source Other Year 2008
al
Tuned
0.207 ± 0.011 0.288 ± 0.018 0.218 ± 0.014 0.161 ± 0.010 0.203 ± 0.009 0.174 ± 0.011 0.195 ± 0.011 0.198 ± 0.007 0.266 ± 0.019 0.173 ± 0.010 0.172 ± 0.012
Test Error (1-AUC)
Default
Op tim
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
101
102
103
Train time (s)
104
Figure G.80: kdd_cup_09_appetency: per-method test error (left) and HPO Pareto trajectory (right).
0.249 ± 0.031 0.248 ± 0.014 0.247 ± 0.027 0.230 ± 0.027 0.235 ± 0.028 0.231 ± 0.023 0.230 ± 0.027 0.238 ± 0.031 -
RandomForest TabDPT ExtraTrees TabM LightGBM TabICLv2 XGBoost RealMLP CatBoost TabPFN-2.6 Metadata Samples 72,983 Features 41 Problem binary Splits 9 Type temporal Missingness 6.4% Domain business & marketing Source Kaggle Year 2011
0.250 0.245 0.240 0.235 0.230
al
0.249 ± 0.031 0.249 ± 0.016 0.250 ± 0.025 0.231 ± 0.027 0.238 ± 0.031 0.234 ± 0.022 0.231 ± 0.027 0.239 ± 0.030 -
0.255
tim
0.254 ± 0.033 0.256 ± 0.018 0.249 ± 0.027 0.232 ± 0.029 0.247 ± 0.022 0.231 ± 0.025 0.237 ± 0.028 0.240 ± 0.030 0.248 ± 0.035 0.228 ± 0.031 0.232 ± 0.032
Dataset: kick
Tuned + Ens.
Test Error (1-AUC)
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Tuned
Op
Default
101
102
103
Train time (s)
104
Figure G.81: kick: per-method test error (left) and HPO Pareto trajectory (right).
0.115 ± 0.018 0.088 ± 0.003 0.093 ± 0.006 0.072 ± 0.005 0.074 ± 0.006 0.072 ± 0.005 0.072 ± 0.006 0.072 ± 0.004 -
ExtraTrees RandomForest TabICLv2 LightGBM CatBoost TabM RealMLP XGBoost Metadata Samples 187,118 Features 206 Problem binary Splits 3 Type temporal Missingness 5.6% Domain business & marketing Source Other Year 2025
0.11 0.10 0.09 0.08 al
0.115 ± 0.019 0.089 ± 0.003 0.094 ± 0.006 0.072 ± 0.005 0.075 ± 0.007 0.072 ± 0.004 0.072 ± 0.005 0.073 ± 0.006 -
0.12
Test Error (1-AUC)
0.115 ± 0.018 0.093 ± 0.005 0.121 ± 0.000 0.073 ± 0.006 0.077 ± 0.006 0.078 ± 0.008 0.076 ± 0.008 0.073 ± 0.004 0.079 ± 0.003
Dataset: kickstarter
Tuned + Ens.
0.07
tim
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Tuned
Op
Default
102
103
104
Train time (s)
105
Figure G.82: kickstarter: per-method test error (left) and HPO Pareto trajectory (right). 77
0.289 ± 0.003 0.254 ± 0.004 0.264 ± 0.004 0.221 ± 0.004 0.226 ± 0.005 0.225 ± 0.005 0.233 ± 0.005 0.223 ± 0.004 0.257 ± 0.003 0.226 ± 0.012 0.213 ± 0.004
0.288 ± 0.003 0.234 ± 0.004 0.233 ± 0.004 0.220 ± 0.004 0.225 ± 0.004 0.223 ± 0.004 0.218 ± 0.004 0.222 ± 0.004 -
0.288 ± 0.003 0.233 ± 0.004 0.231 ± 0.004 0.220 ± 0.004 0.224 ± 0.004 0.222 ± 0.004 0.215 ± 0.004 0.222 ± 0.004 -
Dataset: labour_inspection_complianceTabDPT
RandomForest ExtraTrees TabPFN-2.6 LightGBM XGBoost TabM CatBoost RealMLP TabICLv2 Metadata Samples 63,634 Features 500 Problem binary Splits 9 Type IID Missingness 85.9% Domain industry & manufacturing Source Other Year 2019
0.26 0.25 0.24 0.23 0.22 al
Tuned + Ens.
Test Error (1-AUC)
Tuned
Op tim
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
102
104
103
105
Train time (s)
Figure G.83: labour_inspection_compliance: per-method test error (left) and HPO Pareto trajectory (right).
Tuned
Tuned + Ens.
0.305 0.317 0.312 0.286 0.287 0.285 0.287 0.291 -
0.303 0.313 0.310 0.285 0.286 0.284 0.285 0.289 -
Dataset: lending_club_1m
0.320
0.310 0.305 0.300 0.295 0.290 0.285
TabICLv2 ExtraTrees RandomForest TabM LightGBM RealMLP CatBoost XGBoost Metadata Samples 1,064,751 Features 278 Problem binary Splits 1 Type temporal Missingness 32.3% Domain finance Source Kaggle Year 2018
al
Test Error (1-AUC)
0.315
Op tim
Default Linear 0.306 RandomForest 0.308 ExtraTrees 0.319 CatBoost 0.287 LightGBM 0.288 XGBoost 0.288 RealMLP 0.287 TabM 0.290 TabDPT TabPFN-2.6 TabICLv2 0.315
104
103
Train time (s)
105
Figure G.84: lending_club_1m: per-method test error (left) and HPO Pareto trajectory (right).
0.308 ± 0.049 0.328 ± 0.048 0.315 ± 0.051 0.303 ± 0.048 0.318 ± 0.042 0.306 ± 0.045 0.313 ± 0.046 0.310 ± 0.044 -
Dataset: ljubljana_breast_cancer 0.34 0.33 0.32 0.31 0.30 0.29
RandomForest ExtraTrees LightGBM RealMLP XGBoost TabM TabDPT CatBoost TabICLv2 TabPFN-2.6 Metadata Samples 286 Features 9 Problem binary Splits 60 Type IID Missingness 0.3% Domain medical & healthcare Source UCI Year 1988
al
Tuned + Ens.
0.306 ± 0.048 0.328 ± 0.049 0.313 ± 0.053 0.304 ± 0.051 0.312 ± 0.048 0.307 ± 0.046 0.324 ± 0.046 0.313 ± 0.047 -
Test Error (1-AUC)
Tuned
0.317 ± 0.048 0.332 ± 0.044 0.333 ± 0.047 0.287 ± 0.043 0.319 ± 0.043 0.342 ± 0.044 0.323 ± 0.044 0.304 ± 0.044 0.288 ± 0.048 0.284 ± 0.045 0.284 ± 0.047
tim
Default
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
102
Train time (s)
103
Figure G.85: ljubljana_breast_cancer: per-method test error (left) and HPO Pareto trajectory (right).
1.396 ± 0.078 1.401 ± 0.082 1.401 ± 0.078 1.381 ± 0.068 1.348 ± 0.081 1.395 ± 0.076 1.329 ± 0.082 1.315 ± 0.078 -
Dataset: ljubljana_primary_tumor RandomForest
ExtraTrees XGBoost TabDPT CatBoost LightGBM TabICLv2 TabPFN-2.6 RealMLP TabM Metadata Samples 302 Features 17 Problem multiclass Classes 11 Splits 60 Type IID Missingness 3.9% Domain medical & healthcare Source UCI Year 1987
1.475 1.450 1.425 1.400 1.375 1.350 1.325
al
1.399 ± 0.078 1.404 ± 0.086 1.399 ± 0.081 1.387 ± 0.067 1.355 ± 0.081 1.400 ± 0.083 1.343 ± 0.087 1.320 ± 0.082 -
1.500
tim
1.423 ± 0.105 1.483 ± 0.108 1.493 ± 0.114 1.449 ± 0.063 1.421 ± 0.083 1.460 ± 0.088 1.454 ± 0.087 1.313 ± 0.080 1.391 ± 0.071 1.336 ± 0.084 1.345 ± 0.094
Tuned + Ens.
Test Error (logloss)
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Tuned
Op
Default
102
Train time (s)
103
Figure G.86: ljubljana_primary_tumor: per-method test error (left) and HPO Pareto trajectory (right). 78
0.153 ± 0.078 0.239 ± 0.045 0.182 ± 0.030 0.219 ± 0.079 0.168 ± 0.054 0.213 ± 0.057 0.155 ± 0.087 0.143 ± 0.065 -
0.140 ± 0.067 0.240 ± 0.045 0.182 ± 0.030 0.217 ± 0.078 0.168 ± 0.054 0.213 ± 0.057 0.148 ± 0.078 0.140 ± 0.062 -
RandomForest CatBoost XGBoost ExtraTrees TabICLv2 LightGBM RealMLP TabDPT TabM TabPFN-2.6 Metadata Samples 197 Features 12,600 Problem multiclass Classes 4 Splits 60 Type IID Missingness 0.0% Domain medical & healthcare Source Other Year 2001
0.300 0.275 0.250 0.225 0.200 0.175 0.150 0.125
al
0.143 ± 0.072 0.307 ± 0.031 0.326 ± 0.028 0.224 ± 0.086 0.191 ± 0.050 0.229 ± 0.057 0.150 ± 0.078 0.133 ± 0.064 0.144 ± 0.056 0.127 ± 0.057 0.172 ± 0.080
Dataset: lung_cancer
0.325
tim
Tuned + Ens.
Test Error (logloss)
Tuned
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
103
104
Train time (s)
105
Figure G.87: lung_cancer: per-method test error (left) and HPO Pareto trajectory (right).
0.224 ± 0.052 0.258 ± 0.057 0.265 ± 0.055 0.252 ± 0.050 0.241 ± 0.055 0.247 ± 0.060 0.245 ± 0.054 0.249 ± 0.053 -
0.224 ± 0.051 0.256 ± 0.058 0.263 ± 0.055 0.251 ± 0.050 0.240 ± 0.054 0.244 ± 0.059 0.246 ± 0.050 0.249 ± 0.053 -
ExtraTrees RandomForest CatBoost RealMLP TabM XGBoost TabPFN-2.6 LightGBM TabDPT Metadata Samples 187 Features 22,215 Problem binary Splits 60 Type IID Missingness 0.0% Domain medical & healthcare Source GOV Website Year 2006
0.27 0.26 0.25 0.24 0.23 al
0.223 ± 0.052 0.266 ± 0.053 0.265 ± 0.051 0.249 ± 0.053 0.242 ± 0.059 0.250 ± 0.062 0.246 ± 0.053 0.246 ± 0.053 0.223 ± 0.054 0.241 ± 0.052 0.273 ± 0.053
Dataset: lung_cancer_epithelial_genexpTabICLv2 Test Error (1-AUC)
Tuned + Ens.
tim
Tuned
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
103
104
Train time (s)
105
Figure G.88: lung_cancer_epithelial_genexp: per-method test error (left) and HPO Pareto trajectory (right).
Tuned + Ens.
0.163 0.162 0.162 0.157 0.157 0.157 0.157 0.158 -
0.163 0.162 0.162 0.157 0.157 0.157 0.157 0.158 -
Dataset: maps_router_eta_1m 0.166 0.164 0.162 0.160 0.158
TabICLv2 RandomForest ExtraTrees TabM XGBoost CatBoost RealMLP LightGBM Metadata Samples 1,250,000 Features 997 Problem regression Splits 1 Type temporal Missingness 4.7% Domain industry & manufacturing Source Kaggle Year 2024
0.156
tim
al
Test Error (RMSE)
Linear 0.163 RandomForest 0.162 ExtraTrees 0.162 CatBoost 0.157 LightGBM 0.157 XGBoost 0.157 RealMLP 0.158 TabM 0.158 TabDPT TabPFN-2.6 TabICLv2 0.167
Tuned
Op
Default
104
105
Train time (s)
Figure G.89: maps_router_eta_1m: per-method test error (left) and HPO Pareto trajectory (right).
0.096 ± 0.012 0.117 ± 0.014 0.115 ± 0.012 0.096 ± 0.011 0.090 ± 0.011 0.097 ± 0.010 0.091 ± 0.012 0.087 ± 0.009 -
Dataset: marketing_campaign 0.11 0.10 0.09 0.08
RandomForest ExtraTrees XGBoost TabDPT CatBoost RealMLP LightGBM TabM TabPFN-2.6 TabICLv2 Metadata Samples 2,240 Features 34 Problem binary Splits 30 Type IID Missingness 0.0% Domain business & marketing Source Kaggle Year 2020
al
Tuned + Ens.
0.097 ± 0.013 0.116 ± 0.013 0.118 ± 0.013 0.097 ± 0.012 0.090 ± 0.011 0.098 ± 0.010 0.093 ± 0.012 0.091 ± 0.009 -
Test Error (1-AUC)
Tuned
0.096 ± 0.012 0.115 ± 0.013 0.117 ± 0.013 0.095 ± 0.011 0.100 ± 0.012 0.103 ± 0.011 0.096 ± 0.012 0.097 ± 0.011 0.096 ± 0.014 0.079 ± 0.010 0.076 ± 0.010
tim
Default
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
101
102
Train time (s)
103
Figure G.90: marketing_campaign: per-method test error (left) and HPO Pareto trajectory (right). 79
0.789 ± 0.026 0.468 ± 0.038 0.455 ± 0.048 0.452 ± 0.033 0.458 ± 0.034 0.455 ± 0.032 0.436 ± 0.040 0.472 ± 0.043 -
0.782 ± 0.023 0.436 ± 0.037 0.426 ± 0.034 0.449 ± 0.032 0.465 ± 0.029 0.453 ± 0.033 0.424 ± 0.034 0.466 ± 0.038 -
0.525 0.500 0.475 0.450 0.425 0.400 0.375
al
0.795 ± 0.022 0.452 ± 0.043 0.457 ± 0.038 0.462 ± 0.033 0.472 ± 0.037 0.456 ± 0.036 0.544 ± 0.041 0.510 ± 0.038 0.384 ± 0.041 0.398 ± 0.044 0.377 ± 0.040
Dataset: maternal_health_risk
0.550
tim
Tuned + Ens.
Test Error (logloss)
Tuned
TabM LightGBM XGBoost CatBoost RandomForest ExtraTrees RealMLP TabPFN-2.6 TabDPT TabICLv2 Metadata Samples 1,014 Features 6 Problem multiclass Classes 3 Splits 30 Type IID Missingness 0.0% Domain medical & healthcare Source UCI Year 2020
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
101
102
103
Train time (s)
Figure G.91: maternal_health_risk: per-method test error (left) and HPO Pareto trajectory (right).
Tuned + Ens.
0.648 0.512 0.516 0.467 0.466 0.472 0.451 0.469 -
0.648 0.512 0.516 0.468 0.470 0.469 0.452 0.467 -
Dataset: mercari_price_suggestion_1mExtraTrees
0.52
RandomForest LightGBM XGBoost CatBoost TabM TabICLv2 RealMLP Metadata Samples 1,250,000 Features 130 Problem regression Splits 1 Type IID Missingness 8.1% Domain business & marketing Source Kaggle Year 2018
Test Error (RMSE)
0.51 0.50 0.49 0.48 0.47
al
0.46 0.45
tim
Linear 0.648 RandomForest 0.512 ExtraTrees 0.518 CatBoost 0.490 LightGBM 0.481 XGBoost 0.472 RealMLP 0.473 TabM 0.473 TabDPT TabPFN-2.6 TabICLv2 0.464
Tuned
Op
Default
104
Train time (s)
105
Figure G.92: mercari_price_suggestion_1m: per-method test error (left) and HPO Pareto trajectory (right).
7.902 ± 1.162 8.541 ± 1.177 7.906 ± 1.147 7.743 ± 1.211 7.746 ± 1.235 7.797 ± 1.243 7.748 ± 1.171 7.765 ± 1.285 -
7.871 ± 1.136 8.574 ± 1.217 7.903 ± 1.158 7.748 ± 1.226 7.724 ± 1.216 7.875 ± 1.290 7.742 ± 1.185 7.724 ± 1.253 -
ExtraTrees TabDPT XGBoost RealMLP CatBoost TabM LightGBM TabICLv2 TabPFN-2.6
8.8 8.6 8.4
Metadata Samples 4,204 Features 371 Problem regression Splits 9 Type temporal Missingness 0.0% Domain industry & manufacturing Source Kaggle Year 2017
8.2 8.0 7.8 7.6
al
8.220 ± 1.240 8.918 ± 1.497 8.548 ± 1.138 7.725 ± 1.215 7.855 ± 1.267 8.428 ± 1.583 7.813 ± 1.214 7.764 ± 1.187 7.889 ± 1.121 7.614 ± 1.234 7.671 ± 1.136
Dataset: mercedes_benz_greener_manufact RandomForest
tim
Tuned + Ens.
Op
Tuned
Test Error (RMSE)
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
101
102
Train time (s)
103
Figure G.93: mercedes_benz_greener_manufacturing: per-method test error (left) and HPO Pareto trajectory (right).
0.264 ± 0.003 0.156 ± 0.003 0.150 ± 0.003 0.134 ± 0.002 0.137 ± 0.003 0.138 ± 0.002 0.133 ± 0.003 0.133 ± 0.003 -
Dataset: miami_housing
0.160
RandomForest ExtraTrees XGBoost LightGBM CatBoost TabM RealMLP TabDPT TabPFN-2.6 TabICLv2 Metadata Samples 13,776 Features 15 Problem regression Splits 9 Type IID Missingness 0.0% Domain finance Source Kaggle Year 2016
0.155 0.150 0.145 0.140 0.135 0.130
al
Tuned + Ens.
0.269 ± 0.003 0.157 ± 0.004 0.150 ± 0.003 0.135 ± 0.002 0.139 ± 0.003 0.139 ± 0.003 0.134 ± 0.003 0.135 ± 0.003 -
tim
Tuned
0.269 ± 0.003 0.159 ± 0.004 0.156 ± 0.003 0.135 ± 0.002 0.141 ± 0.002 0.143 ± 0.002 0.139 ± 0.003 0.136 ± 0.002 0.131 ± 0.003 0.129 ± 0.003 0.128 ± 0.003
Test Error (RMSE)
Default
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
101
102
Train time (s)
103
104
Figure G.94: miami_housing: per-method test error (left) and HPO Pareto trajectory (right). 80
0.479 ± 0.025 0.469 ± 0.017 0.471 ± 0.017 0.451 ± 0.017 0.457 ± 0.018 0.438 ± 0.019 0.434 ± 0.018 0.436 ± 0.018 -
TabDPT ExtraTrees RandomForest LightGBM CatBoost TabICLv2 XGBoost TabM RealMLP TabPFN-2.6 Metadata Samples 1,699 Features 111 Problem multiclass Classes 8 Splits 30 Type IID Missingness 8.5% Domain medical & healthcare Source UCI Year 2020
0.50 0.49 0.48 0.47 0.46 0.45 0.44 al
0.480 ± 0.025 0.478 ± 0.017 0.491 ± 0.020 0.455 ± 0.017 0.460 ± 0.021 0.438 ± 0.019 0.433 ± 0.020 0.439 ± 0.019 -
Dataset: mic
0.51
Test Error (logloss)
0.585 ± 0.038 0.477 ± 0.017 0.498 ± 0.012 0.453 ± 0.018 0.500 ± 0.016 0.469 ± 0.018 0.490 ± 0.036 0.438 ± 0.018 0.508 ± 0.014 0.431 ± 0.021 0.438 ± 0.022
Tuned + Ens.
0.43
tim
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Tuned
Op
Default
101
102
103
Train time (s)
Figure G.95: mic: per-method test error (left) and HPO Pareto trajectory (right).
1.234 ± 0.304 1.259 ± 0.138 1.226 ± 0.115 1.323 ± 0.133 1.294 ± 0.134 1.504 ± 0.140 1.196 ± 0.161 1.121 ± 0.146 -
1.213 ± 0.300 1.237 ± 0.137 1.225 ± 0.113 1.321 ± 0.128 1.295 ± 0.135 1.505 ± 0.140 1.183 ± 0.152 1.118 ± 0.141 -
TabDPT CatBoost LightGBM TabPFN-2.6 RandomForest ExtraTrees RealMLP TabM TabICLv2 Metadata Samples 1,080 Features 76 Problem multiclass Classes 8 Splits 60 Type grouped Missingness 1.7% Domain biology & life sciences Source UCI Year 2015
1.6 1.5 1.4 1.3 1.2 1.1 1.0 0.9
al
1.265 ± 0.347 1.240 ± 0.102 1.300 ± 0.081 1.372 ± 0.120 1.683 ± 0.120 1.669 ± 0.114 1.343 ± 0.194 1.153 ± 0.176 1.352 ± 0.158 1.275 ± 0.101 0.910 ± 0.190
Dataset: mice_protein_trisomy_discriminXGBoost 1.7 Test Error (logloss)
Tuned + Ens.
tim
Tuned
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
102
104
103
Train time (s)
Figure G.96: mice_protein_trisomy_discriminant: per-method test error (left) and HPO Pareto trajectory (right).
Tuned
Tuned + Ens.
0.413 ± 0.104 0.843 ± 0.039 0.990 ± 0.033 0.538 ± 0.087 0.698 ± 0.085 0.897 ± 0.090 0.690 ± 0.070 0.428 ± 0.076 1.184 ± 0.184 0.432 ± 0.087 0.563 ± 0.107
0.415 ± 0.108 0.702 ± 0.083 0.716 ± 0.058 0.483 ± 0.082 0.512 ± 0.087 0.787 ± 0.091 0.613 ± 0.069 0.429 ± 0.076 -
0.412 ± 0.108 0.689 ± 0.078 0.716 ± 0.058 0.484 ± 0.082 0.518 ± 0.082 0.787 ± 0.090 0.611 ± 0.064 0.429 ± 0.074 -
Dataset: micro_mass
1.2
Test Error (logloss)
Default
TabDPT XGBoost ExtraTrees RandomForest RealMLP TabICLv2 LightGBM CatBoost TabPFN-2.6 TabM Metadata Samples 571 Features 1,082 Problem multiclass Classes 20 Splits 60 Type grouped Missingness 0.0% Domain biology & life sciences Source UCI Year 2013
1.1 1.0 0.9 0.8 0.7 0.6 0.5 0.4
Op tim al
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
102
103
Train time (s)
104
105
Figure G.97: micro_mass: per-method test error (left) and HPO Pareto trajectory (right).
Tuned + Ens.
0.175 ± 0.130 0.168 ± 0.158 0.168 ± 0.142 0.218 ± 0.174 0.159 ± 0.150 0.173 ± 0.157 0.150 ± 0.121 0.119 ± 0.119 -
0.179 ± 0.130 0.166 ± 0.146 0.167 ± 0.142 0.220 ± 0.182 0.157 ± 0.149 0.168 ± 0.156 0.145 ± 0.116 0.119 ± 0.119 -
Dataset: musk
CatBoost XGBoost TabPFN-2.6 LightGBM ExtraTrees TabDPT RandomForest RealMLP TabICLv2 TabM Metadata Samples 6,598 Features 166 Problem binary Splits 60 Type grouped Missingness 0.0% Domain chemistry & material science Source UCI Year 1994
0.22
Test Error (1-AUC)
Tuned
0.172 ± 0.123 0.144 ± 0.137 0.154 ± 0.131 0.234 ± 0.180 0.201 ± 0.151 0.193 ± 0.156 0.143 ± 0.128 0.129 ± 0.121 0.152 ± 0.114 0.160 ± 0.125 0.126 ± 0.111
0.20 0.18 0.16 0.14 0.12
tim al
Default
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
102
103
Train time (s)
104
Figure G.98: musk: per-method test error (left) and HPO Pareto trajectory (right). 81
5.312 ± 0.312 5.009 ± 0.380 5.076 ± 0.387 4.961 ± 0.365 4.919 ± 0.372 4.872 ± 0.354 4.812 ± 0.357 4.854 ± 0.345 -
TabDPT ExtraTrees RandomForest CatBoost LightGBM XGBoost TabM RealMLP TabPFN-2.6 TabICLv2 Metadata Samples 793 Features 105 Problem regression Splits 30 Type IID Missingness 0.0% Domain finance Source Kaggle Year 2023
5.3 5.2 5.1 5.0 4.9 4.8 4.7 4.6
al
5.447 ± 0.274 4.967 ± 0.365 5.093 ± 0.388 4.967 ± 0.364 5.022 ± 0.374 4.986 ± 0.348 4.870 ± 0.354 4.872 ± 0.352 -
Dataset: mutual_funds_india
5.4
Test Error (RMSE)
5.457 ± 0.258 5.249 ± 0.379 5.210 ± 0.396 5.245 ± 0.396 5.143 ± 0.382 4.984 ± 0.400 4.914 ± 0.369 5.085 ± 0.328 5.377 ± 0.391 4.605 ± 0.379 4.559 ± 0.388
Tuned + Ens.
tim
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Tuned
Op
Default
101
102
103
Train time (s)
Figure G.99: mutual_funds_india: per-method test error (left) and HPO Pareto trajectory (right).
0.019 ± 0.002 0.020 ± 0.002 0.017 ± 0.002 0.014 ± 0.002 0.014 ± 0.002 0.016 ± 0.002 0.014 ± 0.002 0.014 ± 0.002 -
0.019 ± 0.002 0.019 ± 0.002 0.017 ± 0.002 0.014 ± 0.002 0.014 ± 0.002 0.015 ± 0.002 0.014 ± 0.002 0.014 ± 0.002 -
ExtraTrees TabDPT XGBoost LightGBM TabPFN-2.6 CatBoost TabM RealMLP TabICLv2 Metadata Samples 7,491 Features 85 Problem binary Splits 9 Type IID Missingness 0.0% Domain technology & internet Source UCI Year 2021
0.020 0.018 0.016 0.014 al
0.019 ± 0.002 0.021 ± 0.002 0.021 ± 0.002 0.014 ± 0.002 0.015 ± 0.002 0.015 ± 0.002 0.015 ± 0.002 0.014 ± 0.002 0.016 ± 0.002 0.014 ± 0.002 0.012 ± 0.002
Dataset: naticusdroid_android_permissioRandomForest
0.012
tim
Tuned + Ens.
Test Error (1-AUC)
Tuned
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
101
102
Train time (s)
103
Figure G.100: naticusdroid_android_permissions_dataset: per-method test error (left) and HPO Pareto trajectory (right).
4.364 ± 0.276 4.412 ± 0.266 4.406 ± 0.260 4.395 ± 0.273 4.374 ± 0.280 4.423 ± 0.270 4.362 ± 0.280 4.366 ± 0.277 -
Dataset: obesity_estimation
4.55 4.50 4.45 4.40 4.35 4.30
XGBoost RandomForest ExtraTrees CatBoost LightGBM RealMLP TabM TabPFN-2.6 TabICLv2 TabDPT Metadata Samples 498 Features 14 Problem regression Splits 60 Type IID Missingness 0.0% Domain medical & healthcare Source UCI Year 2019
al
Tuned + Ens.
4.365 ± 0.280 4.406 ± 0.265 4.408 ± 0.261 4.397 ± 0.274 4.385 ± 0.288 4.422 ± 0.269 4.375 ± 0.283 4.371 ± 0.277 -
Test Error (RMSE)
Tuned
4.440 ± 0.280 4.548 ± 0.253 4.489 ± 0.251 4.393 ± 0.276 4.454 ± 0.267 4.508 ± 0.271 4.358 ± 0.281 4.350 ± 0.276 4.280 ± 0.267 4.341 ± 0.284 4.324 ± 0.279
tim
Default
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
102
Train time (s)
103
Figure G.101: obesity_estimation: per-method test error (left) and HPO Pareto trajectory (right).
Default
Tuned
Tuned + Ens.
0.088 ± 0.006 0.073 ± 0.005 0.080 ± 0.006 0.066 ± 0.003 0.065 ± 0.004 0.064 ± 0.004 0.071 ± 0.004 0.065 ± 0.004 0.070 ± 0.005 0.062 ± 0.004 0.063 ± 0.004
0.088 ± 0.006 0.070 ± 0.004 0.070 ± 0.005 0.066 ± 0.004 0.064 ± 0.003 0.065 ± 0.004 0.067 ± 0.003 0.064 ± 0.004 -
0.083 ± 0.006 0.069 ± 0.004 0.070 ± 0.005 0.065 ± 0.003 0.064 ± 0.003 0.064 ± 0.003 0.066 ± 0.004 0.064 ± 0.004 -
Dataset: online_shoppers_purchasing_intTabDPT
ExtraTrees RandomForest RealMLP CatBoost LightGBM TabM XGBoost TabICLv2 TabPFN-2.6 Metadata Samples 12,330 Features 17 Problem binary Splits 9 Type IID Missingness 0.0% Domain business & marketing Source UCI Year 2017
0.0775 0.0750 0.0725 0.0700 0.0675
tim
0.0625
al
0.0650 Op
Test Error (1-AUC)
0.0800
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
101
102
Train time (s)
103
Figure G.102: online_shoppers_purchasing_intention_dataset: per-method test error (left) and HPO Pareto trajectory (right). 82
0.754 ± 0.005 0.587 ± 0.003 0.628 ± 0.003 0.475 ± 0.005 0.476 ± 0.005 0.482 ± 0.004 0.474 ± 0.005 0.445 ± 0.005 0.494 ± 0.006 0.463 ± 0.003 0.421 ± 0.005
0.640 ± 0.004 0.543 ± 0.004 0.545 ± 0.004 0.469 ± 0.005 0.463 ± 0.005 0.461 ± 0.004 0.449 ± 0.005 0.445 ± 0.005 -
0.625 ± 0.003 0.546 ± 0.004 0.543 ± 0.003 0.466 ± 0.005 0.460 ± 0.005 0.461 ± 0.005 0.440 ± 0.004 0.441 ± 0.004 -
Dataset: otto_group_product_classificat RandomForest Test Error (logloss)
Tuned + Ens.
ExtraTrees TabDPT CatBoost TabPFN-2.6 XGBoost LightGBM TabM RealMLP TabICLv2 Metadata Samples 61,878 Features 93 Problem multiclass Classes 9 Splits 9 Type IID Missingness 0.0% Domain business & marketing Source Kaggle Year 2015
0.60 0.55 0.50 0.45 tim al
Tuned
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
102
104
103
Train time (s)
Figure G.103: otto_group_product_classification_challenge: per-method test error (left) and HPO Pareto trajectory (right).
0.383 ± 0.090 0.507 ± 0.087 0.502 ± 0.090 0.491 ± 0.072 0.500 ± 0.085 0.513 ± 0.099 0.465 ± 0.082 0.483 ± 0.072 -
0.381 ± 0.089 0.507 ± 0.087 0.506 ± 0.092 0.488 ± 0.074 0.503 ± 0.084 0.514 ± 0.096 0.458 ± 0.081 0.479 ± 0.072 -
TabICLv2 LightGBM CatBoost RandomForest TabM ExtraTrees RealMLP TabPFN-2.6 TabDPT Metadata Samples 181 Features 6,771 Problem binary Splits 60 Type grouped Missingness 0.0% Domain medical & healthcare Source Other Year 2003
0.52 0.50 0.48 0.46 0.44 0.42 0.40
al
0.373 ± 0.090 0.482 ± 0.061 0.465 ± 0.063 0.497 ± 0.079 0.512 ± 0.093 0.524 ± 0.110 0.452 ± 0.077 0.475 ± 0.075 0.384 ± 0.099 0.404 ± 0.091 0.511 ± 0.072
Dataset: pancreatic_cancer_mouse_detectXGBoost
0.38
tim
Tuned + Ens.
Test Error (1-AUC)
Tuned
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
102
104
103
Train time (s)
Figure G.104: pancreatic_cancer_mouse_detection: per-method test error (left) and HPO Pareto trajectory (right).
0.221 ± 0.131 0.179 ± 0.126 0.154 ± 0.115 0.166 ± 0.120 0.166 ± 0.128 0.209 ± 0.135 0.126 ± 0.104 0.157 ± 0.089 -
0.227 ± 0.132 0.181 ± 0.125 0.155 ± 0.117 0.167 ± 0.119 0.170 ± 0.131 0.207 ± 0.135 0.122 ± 0.100 0.152 ± 0.090 -
XGBoost TabPFN-2.6 LightGBM TabICLv2 CatBoost RandomForest ExtraTrees TabM RealMLP Metadata Samples 195 Features 23 Problem binary Splits 60 Type grouped Missingness 0.0% Domain medical & healthcare Source UCI Year 2007
0.22 0.20 0.18 0.16 0.14 al
0.200 ± 0.116 0.159 ± 0.113 0.144 ± 0.114 0.177 ± 0.125 0.213 ± 0.156 0.213 ± 0.135 0.211 ± 0.141 0.134 ± 0.089 0.232 ± 0.156 0.182 ± 0.126 0.169 ± 0.138
Dataset: parkinsons_biomedical_voice_meTabDPT
0.12
tim
Tuned + Ens.
Test Error (1-AUC)
Tuned
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
102
Train time (s)
103
Figure G.105: parkinsons_biomedical_voice_measurements: per-method test error (left) and HPO Pareto trajectory (right).
5.185 ± 0.027 3.481 ± 0.027 3.469 ± 0.024 3.434 ± 0.026 3.448 ± 0.022 3.408 ± 0.021 3.185 ± 0.020 3.290 ± 0.028 -
5.143 ± 0.027 3.476 ± 0.026 3.470 ± 0.024 3.405 ± 0.020 3.453 ± 0.022 3.409 ± 0.021 3.106 ± 0.024 3.247 ± 0.023 -
ExtraTrees LightGBM XGBoost CatBoost TabM RealMLP TabPFN-2.6 TabICLv2 TabDPT
3.5
Test Error (RMSE)
5.195 ± 0.028 3.589 ± 0.027 3.577 ± 0.023 3.525 ± 0.019 3.485 ± 0.022 3.520 ± 0.022 3.449 ± 0.038 3.419 ± 0.020 2.898 ± 0.022 3.011 ± 0.023 2.944 ± 0.023
Dataset: physiochemical_protein RandomForest
3.6
3.4 3.3
Metadata Samples 45,730 Features 9 Problem regression Splits 9 Type IID Missingness 0.0% Domain chemistry & material science Source UCI Year 2013
3.2 3.1 3.0 2.9
al
Tuned + Ens.
tim
Tuned
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
102
103
Train time (s)
104
Figure G.106: physiochemical_protein: per-method test error (left) and HPO Pareto trajectory (right). 83
0.112 ± 0.013 0.069 ± 0.013 0.107 ± 0.010 0.042 ± 0.008 0.045 ± 0.010 0.043 ± 0.009 0.037 ± 0.008 0.031 ± 0.007 -
0.105 ± 0.010 0.065 ± 0.011 0.107 ± 0.008 0.041 ± 0.008 0.045 ± 0.011 0.044 ± 0.009 0.037 ± 0.009 0.033 ± 0.008 -
ExtraTrees RandomForest LightGBM XGBoost CatBoost RealMLP TabDPT TabM TabICLv2 TabPFN-2.6 Metadata Samples 5,790 Features 64 Problem binary Splits 9 Type IID Missingness 1.2% Domain finance Source UCI Year 2010
0.10 0.08 0.06 0.04 0.02
al
0.134 ± 0.014 0.074 ± 0.009 0.121 ± 0.012 0.041 ± 0.008 0.048 ± 0.010 0.043 ± 0.010 0.043 ± 0.010 0.052 ± 0.009 0.034 ± 0.008 0.014 ± 0.006 0.018 ± 0.007
Dataset: polish_companies_bankruptcy 0.12
tim
Tuned + Ens.
Test Error (1-AUC)
Tuned
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
101
102
Train time (s)
103
Figure G.107: polish_companies_bankruptcy: per-method test error (left) and HPO Pareto trajectory (right).
0.370 ± 0.001 0.373 ± 0.004 0.374 ± 0.004 0.357 ± 0.000 0.360 ± 0.000 0.359 ± 0.000 0.357 ± 0.001 0.357 ± 0.001 -
Dataset: porto_seguro
ExtraTrees RandomForest TabICLv2 LightGBM XGBoost TabM CatBoost RealMLP Metadata Samples 595,206 Features 37 Problem binary Splits 3 Type IID Missingness 3.8% Domain insurance Source Kaggle Year 2017
0.380 0.375 0.370 0.365 0.360 al
Tuned + Ens.
0.369 ± 0.001 0.379 ± 0.004 0.380 ± 0.003 0.357 ± 0.000 0.360 ± 0.001 0.360 ± 0.002 0.358 ± 0.001 0.358 ± 0.001 -
tim
Tuned
0.369 ± 0.001 0.383 ± 0.002 0.379 ± 0.002 0.359 ± 0.001 0.369 ± 0.001 0.359 ± 0.001 0.361 ± 0.001 0.359 ± 0.001 0.366 ± 0.001
Test Error (1-AUC)
Default
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
101
102
103
Train time (s)
104
105
Figure G.108: porto_seguro: per-method test error (left) and HPO Pareto trajectory (right).
0.562 ± 0.015 0.578 ± 0.017 0.573 ± 0.016 0.541 ± 0.019 0.542 ± 0.020 0.552 ± 0.018 0.542 ± 0.019 0.545 ± 0.018 -
0.559 ± 0.014 0.574 ± 0.016 0.567 ± 0.016 0.540 ± 0.018 0.541 ± 0.018 0.550 ± 0.017 0.537 ± 0.017 0.541 ± 0.017 -
ExtraTrees TabDPT XGBoost LightGBM TabM CatBoost RealMLP TabPFN-2.6 TabICLv2 Metadata Samples 4,424 Features 36 Problem multiclass Classes 3 Splits 9 Type IID Missingness 0.0% Domain education Source UCI Year 2021
0.58 0.57 0.56 0.55 0.54 al
0.567 ± 0.016 0.583 ± 0.014 0.589 ± 0.013 0.551 ± 0.016 0.554 ± 0.018 0.556 ± 0.021 0.553 ± 0.017 0.545 ± 0.017 0.555 ± 0.020 0.533 ± 0.019 0.531 ± 0.017
Dataset: predict_students_dropout_and_a RandomForest
0.59
0.53
tim
Tuned + Ens.
Test Error (logloss)
Tuned
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
101
102
103
Train time (s)
Figure G.109: predict_students_dropout_and_academic_success: per-method test error (left) and HPO Pareto trajectory (right).
0.027 ± 0.016 0.028 ± 0.015 0.020 ± 0.011 0.018 ± 0.011 0.017 ± 0.011 0.025 ± 0.015 0.018 ± 0.015 0.022 ± 0.019 -
0.025 ± 0.016 0.022 ± 0.012 0.014 ± 0.008 0.018 ± 0.012 0.017 ± 0.011 0.025 ± 0.015 0.017 ± 0.015 0.022 ± 0.019 -
XGBoost TabM RandomForest TabDPT CatBoost LightGBM RealMLP ExtraTrees TabPFN-2.6 Metadata Samples 322 Features 15,154 Problem binary Splits 60 Type IID Missingness 0.0% Domain medical & healthcare Source Other Year 2002
0.05 0.04 0.03 0.02 0.01
al
0.024 ± 0.016 0.026 ± 0.014 0.026 ± 0.015 0.020 ± 0.013 0.023 ± 0.014 0.032 ± 0.017 0.021 ± 0.016 0.025 ± 0.021 0.022 ± 0.011 0.007 ± 0.005 0.056 ± 0.025
Dataset: prostate_cancer_detection TabICLv2 Test Error (1-AUC)
Tuned + Ens.
tim
Tuned
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
103
104
Train time (s)
105
Figure G.110: prostate_cancer_detection: per-method test error (left) and HPO Pareto trajectory (right). 84
0.407 ± 0.007 0.390 ± 0.006 0.387 ± 0.005 0.369 ± 0.006 0.368 ± 0.007 0.369 ± 0.006 0.372 ± 0.004 0.374 ± 0.005 -
0.404 ± 0.006 0.389 ± 0.007 0.385 ± 0.006 0.369 ± 0.006 0.368 ± 0.006 0.369 ± 0.006 0.372 ± 0.005 0.374 ± 0.005 -
TabPFN-2.6 TabICLv2 RandomForest ExtraTrees TabM RealMLP CatBoost XGBoost LightGBM Metadata Samples 144,095 Features 1,799 Problem binary Splits 9 Type IID Missingness 16.7% Domain business & marketing Source Other Year 1997
0.41 0.40 0.39 0.38 0.37
al
0.409 ± 0.007 0.417 ± 0.004 0.403 ± 0.003 0.370 ± 0.006 0.383 ± 0.006 0.374 ± 0.007 0.373 ± 0.004 0.376 ± 0.004 0.400 ± 0.005 0.399 ± 0.006 0.397 ± 0.003
Dataset: pva_revenue_prediction_kddcup98 TabDPT
tim
Tuned + Ens.
Test Error (1-AUC)
Tuned
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
102
104
103
105
Train time (s)
Figure G.111: pva_revenue_prediction_kddcup98: per-method test error (left) and HPO Pareto trajectory (right).
1.221 ± 0.071 1.120 ± 0.067 1.095 ± 0.067 1.082 ± 0.065 1.117 ± 0.069 1.114 ± 0.067 1.067 ± 0.066 1.122 ± 0.065 -
Dataset: qsar_aquatic_toxicity
1.16 1.14 1.12 1.10 1.08 1.06 1.04
TabM RandomForest LightGBM XGBoost ExtraTrees CatBoost RealMLP TabPFN-2.6 TabICLv2 TabDPT Metadata Samples 546 Features 8 Problem regression Splits 60 Type IID Missingness 0.0% Domain biology & life sciences Source UCI Year 2014
al
Tuned + Ens.
1.219 ± 0.071 1.124 ± 0.068 1.096 ± 0.066 1.083 ± 0.065 1.116 ± 0.073 1.115 ± 0.067 1.068 ± 0.065 1.137 ± 0.068 -
Test Error (RMSE)
Tuned
1.290 ± 0.074 1.140 ± 0.072 1.108 ± 0.069 1.082 ± 0.065 1.134 ± 0.069 1.137 ± 0.075 1.068 ± 0.065 1.158 ± 0.066 1.039 ± 0.069 1.053 ± 0.071 1.050 ± 0.069
tim
Default
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
102
Train time (s)
103
Figure G.112: qsar_aquatic_toxicity: per-method test error (left) and HPO Pareto trajectory (right).
Default
Tuned
Tuned + Ens.
0.092 ± 0.012 0.069 ± 0.010 0.066 ± 0.009 0.069 ± 0.011 0.074 ± 0.011 0.074 ± 0.012 0.072 ± 0.011 0.069 ± 0.011 0.064 ± 0.010 0.063 ± 0.010 0.059 ± 0.009
0.078 ± 0.013 0.069 ± 0.011 0.067 ± 0.011 0.069 ± 0.011 0.068 ± 0.012 0.069 ± 0.010 0.069 ± 0.012 0.066 ± 0.012 -
0.077 ± 0.012 0.071 ± 0.011 0.066 ± 0.010 0.068 ± 0.011 0.068 ± 0.011 0.069 ± 0.011 0.067 ± 0.011 0.064 ± 0.010 -
Dataset: qsar_biodeg
RandomForest XGBoost CatBoost LightGBM RealMLP ExtraTrees TabM TabDPT TabPFN-2.6 TabICLv2 Metadata Samples 1,054 Features 41 Problem binary Splits 30 Type IID Missingness 0.0% Domain biology & life sciences Source UCI Year 2013
Test Error (1-AUC)
0.074 0.072 0.070 0.068 0.066 0.064 0.062
0.058
tim
al
0.060 Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
101
102
Train time (s)
103
Figure G.113: qsar_biodeg: per-method test error (left) and HPO Pareto trajectory (right).
0.947 ± 0.049 0.885 ± 0.050 0.867 ± 0.050 0.874 ± 0.050 0.886 ± 0.048 0.879 ± 0.053 0.863 ± 0.049 0.893 ± 0.052 -
Dataset: qsar_fish_toxicity
0.91 0.90 0.89 0.88 0.87 0.86
TabM LightGBM RandomForest XGBoost CatBoost ExtraTrees RealMLP TabPFN-2.6 TabICLv2 TabDPT Metadata Samples 908 Features 6 Problem regression Splits 30 Type IID Missingness 0.0% Domain biology & life sciences Source UCI Year 2015
al
Tuned + Ens.
0.947 ± 0.049 0.887 ± 0.049 0.871 ± 0.051 0.874 ± 0.052 0.890 ± 0.048 0.881 ± 0.053 0.874 ± 0.050 0.897 ± 0.055 -
Test Error (RMSE)
Tuned
0.947 ± 0.049 0.902 ± 0.053 0.874 ± 0.051 0.876 ± 0.052 0.898 ± 0.045 0.906 ± 0.050 0.874 ± 0.048 0.909 ± 0.050 0.853 ± 0.047 0.857 ± 0.050 0.855 ± 0.052
tim
Default
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
101
102
Train time (s)
103
Figure G.114: qsar_fish_toxicity: per-method test error (left) and HPO Pareto trajectory (right). 85
0.921 ± 0.013 0.772 ± 0.010 0.765 ± 0.013 0.748 ± 0.015 0.731 ± 0.013 0.735 ± 0.012 0.730 ± 0.017 0.734 ± 0.018 -
TabICLv2 RandomForest ExtraTrees CatBoost TabDPT XGBoost TabM TabPFN-2.6 LightGBM RealMLP Metadata Samples 5,741 Features 1,024 Problem regression Splits 9 Type IID Missingness 0.0% Domain chemistry & material science Source OpenML Year 2015
0.78 0.77 0.76 0.75 0.74 al
0.923 ± 0.015 0.771 ± 0.014 0.774 ± 0.014 0.751 ± 0.013 0.728 ± 0.014 0.738 ± 0.012 0.736 ± 0.017 0.738 ± 0.018 -
0.79
Test Error (RMSE)
0.995 ± 0.019 0.789 ± 0.011 0.789 ± 0.011 0.749 ± 0.016 0.748 ± 0.016 0.763 ± 0.016 0.737 ± 0.019 0.739 ± 0.016 0.738 ± 0.020 0.733 ± 0.017 0.792 ± 0.014
Dataset: qsar_tid_11
Tuned + Ens.
0.73
tim
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Tuned
Op
Default
101
102
103
Train time (s)
104
Figure G.115: qsar_tid_11: per-method test error (left) and HPO Pareto trajectory (right).
0.094 ± 0.023 0.079 ± 0.016 0.082 ± 0.014 0.080 ± 0.018 0.092 ± 0.020 0.100 ± 0.020 0.091 ± 0.019 0.088 ± 0.020 0.094 ± 0.019 0.071 ± 0.015 0.069 ± 0.017
0.084 ± 0.020 0.081 ± 0.016 0.083 ± 0.015 0.081 ± 0.018 0.076 ± 0.015 0.084 ± 0.018 0.086 ± 0.021 0.087 ± 0.020 -
0.085 ± 0.020 0.081 ± 0.018 0.082 ± 0.015 0.081 ± 0.017 0.076 ± 0.016 0.084 ± 0.018 0.086 ± 0.021 0.086 ± 0.020 -
Dataset: regensburg_pediatric_appendiciTabDPT 0.100
TabM RealMLP XGBoost ExtraTrees CatBoost RandomForest LightGBM TabPFN-2.6 TabICLv2 Metadata Samples 763 Features 144 Problem binary Splits 30 Type IID Missingness 34.5% Domain medical & healthcare Source Other Year 2021
0.095 0.090 0.085 0.080 0.075 0.070
al
Tuned + Ens.
Test Error (1-AUC)
Tuned
Op tim
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
101
102
103
Train time (s)
Figure G.116: regensburg_pediatric_appendicitis: per-method test error (left) and HPO Pareto trajectory (right).
Default
Tuned
Tuned + Ens.
1450.3 ± 120.2 1129.3 ± 130.2 1197.7 ± 190.7 1059.2 ± 129.4 1027.3 ± 136.4 1048.2 ± 138.4 1029.4 ± 242.0 1110.7 ± 103.4 2151.7 ± 169.8
1450.4 ± 120.5 1128.8 ± 135.7 1183.6 ± 193.1 1021.1 ± 143.5 1007.3 ± 136.9 1011.4 ± 107.5 1006.1 ± 216.9 1090.7 ± 86.2 -
1450.4 ± 120.5 2000 1127.1 ± 135.0 1182.2 ± 191.6 1800 1018.7 ± 131.6 1600 999.0 ± 141.9 1009.2 ± 112.1 1400 993.7 ± 217.1 1200 1021.3 ± 104.4 1000 -
Dataset: rossmann_store_sales
TabICLv2 ExtraTrees RandomForest TabM CatBoost XGBoost LightGBM RealMLP Metadata Samples 844,392 Features 24 Problem regression Splits 3 Type temporal Missingness 14.3% Domain business & marketing Source Kaggle Year 2015
tim
al
Test Error (RMSE)
2200
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
104
103
105
Train time (s)
Figure G.117: rossmann_store_sales: per-method test error (left) and HPO Pareto trajectory (right).
0.184 ± 0.003 0.225 ± 0.008 0.236 ± 0.008 0.162 ± 0.005 0.163 ± 0.005 0.163 ± 0.005 0.160 ± 0.005 0.161 ± 0.005 0.185 ± 0.005 0.158 ± 0.005 0.183 ± 0.011
0.183 ± 0.004 0.169 ± 0.007 0.178 ± 0.005 0.161 ± 0.005 0.161 ± 0.005 0.160 ± 0.005 0.159 ± 0.005 0.159 ± 0.005 -
0.183 ± 0.004 0.167 ± 0.006 0.176 ± 0.005 0.161 ± 0.005 0.160 ± 0.005 0.160 ± 0.005 0.159 ± 0.005 0.159 ± 0.005 -
Dataset: santander_customer_satisfaction TabDPT
TabICLv2 ExtraTrees RandomForest CatBoost LightGBM XGBoost TabM RealMLP TabPFN-2.6 Metadata Samples 71,080 Features 307 Problem binary Splits 9 Type IID Missingness 0.0% Domain business & marketing Source Kaggle Year 2016
0.23
Test Error (1-AUC)
Tuned + Ens.
0.22 0.21 0.20 0.19 0.18 0.17 0.16
tim al
Tuned
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
101
102
103
Train time (s)
104
Figure G.118: santander_customer_satisfaction: per-method test error (left) and HPO Pareto trajectory (right). 86
0.140 ± 0.002 0.141 ± 0.002 0.175 ± 0.002 0.092 ± 0.002 0.091 ± 0.002 0.093 ± 0.002 0.103 ± 0.001 0.103 ± 0.002 -
0.140 ± 0.002 0.143 ± 0.002 0.138 ± 0.002 0.092 ± 0.002 0.091 ± 0.002 0.093 ± 0.002 0.102 ± 0.002 0.102 ± 0.002 -
ExtraTrees TabICLv2 TabM RealMLP XGBoost CatBoost LightGBM Metadata Samples 200,000 Features 600 Problem binary Splits 9 Type IID Missingness 0.0% Domain finance Source Kaggle Year 2019
0.16 0.15 0.14 0.13 0.12 0.11 0.10 0.09
al
0.140 ± 0.002 0.141 ± 0.002 0.166 ± 0.001 0.093 ± 0.002 0.097 ± 0.002 0.099 ± 0.002 0.113 ± 0.002 0.103 ± 0.002 0.116 ± 0.002
Dataset: santander_customer_transaction RandomForest
Tuned + Ens.
Test Error (1-AUC)
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Tuned
Op tim
Default
102
103
104
105
Train time (s)
Figure G.119: santander_customer_transaction_prediction: per-method test error (left) and HPO Pareto trajectory (right).
Tuned + Ens.
1.413 ± 0.017 1.338 ± 0.019 1.345 ± 0.016 1.350 ± 0.015 1.348 ± 0.019 1.343 ± 0.023 1.362 ± 0.021 1.363 ± 0.018 -
1.404 ± 0.015 1.337 ± 0.020 1.344 ± 0.016 1.348 ± 0.016 1.344 ± 0.019 1.342 ± 0.021 1.347 ± 0.019 1.360 ± 0.018 -
Dataset: santander_transaction_value 1.44 1.42 1.40 1.38 1.36 1.34 1.32
TabDPT TabM CatBoost RealMLP LightGBM ExtraTrees XGBoost RandomForest TabPFN-2.6 TabICLv2 Metadata Samples 4,447 Features 540 Problem regression Splits 9 Type IID Missingness 41.7% Domain finance Source Kaggle Year 2018
tim al
Tuned
1.486 ± 0.019 1.341 ± 0.023 1.344 ± 0.017 1.350 ± 0.014 1.349 ± 0.020 1.346 ± 0.020 1.379 ± 0.017 1.363 ± 0.018 1.445 ± 0.022 1.329 ± 0.019 1.323 ± 0.020
Test Error (RMSE)
Default
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
101
102
Train time (s)
104
103
Figure G.120: santander_transaction_value: per-method test error (left) and HPO Pareto trajectory (right).
2.722 ± 0.259 2.138 ± 0.216 2.054 ± 0.177 2.083 ± 0.205 2.058 ± 0.180 2.115 ± 0.202 2.086 ± 0.200 2.083 ± 0.194 -
2.4
Dataset: sat11_hand_algo_runtime TabDPT
RandomForest TabICLv2 XGBoost RealMLP CatBoost TabM LightGBM ExtraTrees TabPFN-2.6 Metadata Samples 2,960 Features 169 Problem regression Splits 30 Type grouped Missingness 3.6% Domain technology & internet Source ASlib Year 2011
2.3 2.2 2.1 2.0
al
Tuned + Ens.
2.666 ± 0.232 2.141 ± 0.217 2.065 ± 0.173 2.098 ± 0.217 2.046 ± 0.181 2.115 ± 0.208 2.105 ± 0.210 2.111 ± 0.195 -
Test Error (RMSE)
Tuned
3.757 ± 0.485 2.189 ± 0.247 2.064 ± 0.183 2.111 ± 0.204 2.236 ± 0.201 2.308 ± 0.228 2.110 ± 0.221 2.101 ± 0.189 2.413 ± 0.224 1.976 ± 0.173 2.136 ± 0.171
tim
Default
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
102
Train time (s)
103
Figure G.121: sat11_hand_algo_runtime: per-method test error (left) and HPO Pareto trajectory (right).
0.277 ± 0.031 0.267 ± 0.019 0.260 ± 0.018 0.254 ± 0.017 0.251 ± 0.018 0.250 ± 0.017 0.247 ± 0.015 0.254 ± 0.013 -
Dataset: sberbank_housing_market_forecaTabICLv2
RandomForest TabDPT ExtraTrees TabM CatBoost LightGBM XGBoost RealMLP TabPFN-2.6 Metadata Samples 27,195 Features 395 Problem regression Splits 5 Type temporal Missingness 4.0% Domain business & marketing Source Kaggle Year 2017
0.28 0.27 0.26 0.25 al
Tuned + Ens.
0.276 ± 0.021 0.273 ± 0.020 0.262 ± 0.019 0.255 ± 0.018 0.253 ± 0.016 0.250 ± 0.018 0.248 ± 0.013 0.255 ± 0.013 -
tim
Tuned
0.579 ± 0.360 0.273 ± 0.021 0.262 ± 0.018 0.252 ± 0.018 0.254 ± 0.016 0.254 ± 0.017 0.252 ± 0.017 0.258 ± 0.015 0.261 ± 0.022 0.243 ± 0.016 0.285 ± 0.037
Test Error (RMSE)
Default
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
102
103
Train time (s)
104
Figure G.122: sberbank_housing_market_forecasting: per-method test error (left) and HPO Pareto trajectory (right). 87
Default
Tuned
Dataset: sdss_17
Tuned + Ens.
RealMLP TabDPT TabM LightGBM ExtraTrees RandomForest CatBoost XGBoost TabICLv2 TabPFN-2.6 Metadata Samples 78,053 Features 11 Problem multiclass Classes 3 Splits 9 Type IID Missingness 0.0% Domain physics & astronomy Source Kaggle Year 2022
0.136 ± 0.003 0.077 ± 0.003 0.081 ± 0.002 0.074 ± 0.002 0.084 ± 0.002 0.073 ± 0.002 0.088 ± 0.002 0.085 ± 0.002 -
0.12 0.11 0.10 0.09 0.08 0.07
al
0.147 ± 0.003 0.077 ± 0.003 0.085 ± 0.002 0.074 ± 0.002 0.088 ± 0.002 0.074 ± 0.002 0.087 ± 0.002 0.085 ± 0.002 -
tim
0.147 ± 0.003 0.085 ± 0.002 0.131 ± 0.002 0.075 ± 0.002 0.088 ± 0.002 0.074 ± 0.002 0.104 ± 0.004 0.097 ± 0.003 0.085 ± 0.002 0.069 ± 0.002 0.071 ± 0.003
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Test Error (logloss)
0.13
101
102
104
103
Train time (s)
Figure G.123: sdss_17: per-method test error (left) and HPO Pareto trajectory (right).
0.238 ± 0.023 0.235 ± 0.025 0.226 ± 0.024 0.228 ± 0.022 0.230 ± 0.021 0.230 ± 0.020 0.232 ± 0.023 0.233 ± 0.024 -
RandomForest TabM RealMLP XGBoost LightGBM TabICLv2 ExtraTrees CatBoost TabDPT TabPFN-2.6
0.245 0.240
Metadata Samples 2,584 Features 15 Problem binary Splits 30 Type IID Missingness 0.0% Domain environmental science & climate Source UCI Year 2013
0.235 0.230 0.225 0.220
al
0.241 ± 0.023 0.236 ± 0.027 0.235 ± 0.026 0.228 ± 0.022 0.231 ± 0.021 0.231 ± 0.020 0.238 ± 0.024 0.234 ± 0.026 -
0.250
tim
0.242 ± 0.024 0.249 ± 0.022 0.251 ± 0.024 0.225 ± 0.022 0.247 ± 0.025 0.240 ± 0.025 0.239 ± 0.026 0.234 ± 0.024 0.222 ± 0.023 0.220 ± 0.024 0.227 ± 0.035
Dataset: seismic_bumps
Tuned + Ens.
Test Error (1-AUC)
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Tuned
Op
Default
101
102
Train time (s)
103
Figure G.124: seismic_bumps: per-method test error (left) and HPO Pareto trajectory (right).
Default
Tuned
Dataset: sepsis_prediction_1m
Tuned + Ens. 0.205
0.280 0.195 0.202 0.188 0.178 0.181 0.182 0.178 -
Test Error (1-AUC)
0.200 0.195 0.190 0.185 0.180
ExtraTrees TabICLv2 RandomForest CatBoost RealMLP XGBoost TabM LightGBM Metadata Samples 1,228,686 Features 42 Problem binary Splits 1 Type grouped Missingness 66.7% Domain medical & healthcare Source Other Year 2019
al
0.280 0.196 0.205 0.188 0.179 0.181 0.183 0.181 -
Op tim
Linear 0.304 RandomForest 0.196 ExtraTrees 0.205 CatBoost 0.190 LightGBM 0.187 XGBoost 0.185 RealMLP 0.186 TabM 0.181 TabDPT TabPFN-2.6 TabICLv2 0.201
102
103
104
Train time (s)
105
Figure G.125: sepsis_prediction_1m: per-method test error (left) and HPO Pareto trajectory (right).
0.294 ± 0.001 0.311 ± 0.002 0.311 ± 0.002 0.294 ± 0.002 0.297 ± 0.002 0.297 ± 0.002 0.293 ± 0.002 0.294 ± 0.002 0.296 ± 0.002 0.293 ± 0.002 0.301 ± 0.007
0.294 ± 0.002 0.294 ± 0.002 0.294 ± 0.001 0.293 ± 0.002 0.293 ± 0.002 0.294 ± 0.001 0.293 ± 0.002 0.294 ± 0.002 -
0.294 ± 0.002 0.294 ± 0.002 0.294 ± 0.002 0.293 ± 0.002 0.293 ± 0.002 0.294 ± 0.002 0.293 ± 0.002 0.293 ± 0.002 -
Dataset: sepsis_survival_minimal_clinicTabICLv2
TabDPT RandomForest XGBoost ExtraTrees TabM CatBoost TabPFN-2.6 LightGBM RealMLP Metadata Samples 110,204 Features 3 Problem binary Splits 9 Type IID Missingness 0.0% Domain medical & healthcare Source UCI Year 2020
0.3100 0.3075 0.3050 0.3025 0.3000 0.2975 0.2950 0.2925
tim al
Tuned + Ens.
Op
Tuned
Test Error (1-AUC)
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
101
102
103
Train time (s)
104
Figure G.126: sepsis_survival_minimal_clinical_records: per-method test error (left) and HPO Pareto trajectory (right). 88
1.420 ± 0.067 1.342 ± 0.075 1.360 ± 0.091 1.315 ± 0.068 1.312 ± 0.069 1.317 ± 0.070 1.332 ± 0.089 1.447 ± 0.210 -
1.409 ± 0.069 1.341 ± 0.074 1.355 ± 0.091 1.315 ± 0.068 1.315 ± 0.071 1.316 ± 0.068 1.334 ± 0.091 1.413 ± 0.148 -
TabM ExtraTrees TabICLv2 RandomForest RealMLP XGBoost LightGBM CatBoost Metadata Samples 116,954 Features 480 Problem regression Splits 6 Type temporal Missingness 25.3% Domain business & marketing Source GOV Website Year 2025
1.42 1.40 1.38 1.36 1.34 1.32
al
1.416 ± 0.062 1.344 ± 0.073 1.360 ± 0.091 1.315 ± 0.068 1.321 ± 0.065 1.323 ± 0.067 1.349 ± 0.098 1.439 ± 0.199 1.348 ± 0.081
Dataset: sf_permit_time
1.44
Test Error (RMSE)
Tuned + Ens.
tim
Tuned
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
102
104
103
105
Train time (s)
Figure G.127: sf_permit_time: per-method test error (left) and HPO Pareto trajectory (right).
0.231 ± 0.036 0.258 ± 0.035 0.240 ± 0.037 0.244 ± 0.036 0.239 ± 0.035 0.261 ± 0.034 0.241 ± 0.036 0.250 ± 0.034 -
0.227 ± 0.036 0.259 ± 0.035 0.241 ± 0.037 0.244 ± 0.035 0.239 ± 0.035 0.260 ± 0.034 0.240 ± 0.036 0.247 ± 0.036 -
RandomForest TabM CatBoost ExtraTrees TabPFN-2.6 LightGBM RealMLP TabICLv2 TabDPT Metadata Samples 462 Features 9 Problem binary Splits 60 Type IID Missingness 0.0% Domain medical & healthcare Source Kaggle Year 1983
0.27 0.26 0.25 0.24 0.23
al
0.228 ± 0.036 0.266 ± 0.032 0.253 ± 0.034 0.246 ± 0.036 0.257 ± 0.033 0.278 ± 0.031 0.239 ± 0.036 0.245 ± 0.036 0.229 ± 0.038 0.240 ± 0.035 0.230 ± 0.037
Dataset: south_africa_coronary_heart_diXGBoost 0.28 Test Error (1-AUC)
Tuned + Ens.
tim
Tuned
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
102
Train time (s)
103
Figure G.128: south_africa_coronary_heart_disease: per-method test error (left) and HPO Pareto trajectory (right).
0.149 ± 0.009 0.177 ± 0.014 0.172 ± 0.014 0.109 ± 0.015 0.105 ± 0.014 0.107 ± 0.015 0.104 ± 0.014 0.111 ± 0.015 -
Dataset: splice
TabDPT RandomForest ExtraTrees TabM CatBoost XGBoost LightGBM RealMLP TabPFN-2.6 TabICLv2 Metadata Samples 3,190 Features 60 Problem multiclass Classes 3 Splits 30 Type IID Missingness 0.0% Domain biology & life sciences Source UCI Year 1991
0.40 0.35 0.30 0.25 0.20 0.15 0.10
al
Tuned + Ens.
0.150 ± 0.010 0.180 ± 0.015 0.174 ± 0.016 0.114 ± 0.015 0.105 ± 0.014 0.107 ± 0.015 0.105 ± 0.014 0.112 ± 0.015 -
Test Error (logloss)
Tuned
0.170 ± 0.013 0.327 ± 0.008 0.403 ± 0.009 0.111 ± 0.014 0.110 ± 0.014 0.117 ± 0.017 0.128 ± 0.019 0.112 ± 0.015 0.200 ± 0.023 0.098 ± 0.016 0.096 ± 0.016
tim
Default
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
101
102
Train time (s)
103
Figure G.129: splice: per-method test error (left) and HPO Pareto trajectory (right).
2.729 ± 0.175 2.676 ± 0.177 2.683 ± 0.170 2.694 ± 0.176 2.694 ± 0.174 2.704 ± 0.181 2.680 ± 0.183 2.682 ± 0.176 -
Dataset: student_portuguese_performance XGBoost
CatBoost LightGBM TabM ExtraTrees RealMLP RandomForest TabDPT TabPFN-2.6 TabICLv2 Metadata Samples 649 Features 30 Problem regression Splits 60 Type IID Missingness 0.0% Domain education Source UCI Year 2008
2.76 2.74 2.72 2.70 2.68 2.66
al
Tuned + Ens.
2.769 ± 0.175 2.676 ± 0.179 2.698 ± 0.171 2.696 ± 0.177 2.707 ± 0.173 2.705 ± 0.182 2.689 ± 0.183 2.683 ± 0.177 -
tim
Tuned
2.768 ± 0.175 2.712 ± 0.174 2.681 ± 0.167 2.699 ± 0.183 2.713 ± 0.178 2.763 ± 0.175 2.687 ± 0.180 2.692 ± 0.175 2.666 ± 0.178 2.661 ± 0.181 2.657 ± 0.190
Test Error (RMSE)
Default
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
102
Train time (s)
103
Figure G.130: student_portuguese_performance: per-method test error (left) and HPO Pareto trajectory (right). 89
17.42 ± 0.09 9.46 ± 0.11 9.34 ± 0.12 9.33 ± 0.13 9.34 ± 0.12 9.31 ± 0.12 9.38 ± 0.15 9.20 ± 0.18 -
9.6
17.26 ± 0.08 9.48 ± 0.10 9.34 ± 0.12 9.31 ± 0.13 9.26 ± 0.12 9.31 ± 0.13 9.22 ± 0.16 9.21 ± 0.17 -
Test Error (RMSE)
17.43 ± 0.09 9.65 ± 0.08 9.46 ± 0.13 9.33 ± 0.14 9.36 ± 0.11 9.41 ± 0.11 9.46 ± 0.17 9.44 ± 0.17 8.93 ± 0.14 9.11 ± 0.16 8.87 ± 0.17
Dataset: superconductivity
Tuned + Ens. 9.5 9.4 9.3 9.2 9.1 9.0 8.9
RandomForest ExtraTrees XGBoost CatBoost LightGBM RealMLP TabM TabPFN-2.6 TabDPT TabICLv2 Metadata Samples 21,263 Features 81 Problem regression Splits 9 Type IID Missingness 0.0% Domain physics & astronomy Source UCI Year 2018
tim al
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Tuned
Op
Default
102
104
103
Train time (s)
Figure G.131: superconductivity: per-method test error (left) and HPO Pareto trajectory (right).
0.063 ± 0.015 0.062 ± 0.012 0.059 ± 0.010 0.054 ± 0.012 0.059 ± 0.015 0.058 ± 0.015 0.060 ± 0.012 0.056 ± 0.012 0.056 ± 0.010 0.048 ± 0.009 0.051 ± 0.009
0.060 ± 0.015 0.062 ± 0.012 0.059 ± 0.010 0.054 ± 0.012 0.052 ± 0.012 0.054 ± 0.013 0.057 ± 0.013 0.054 ± 0.012 -
Dataset: taiwanese_bankruptcy_prediction RandomForest
Tuned + Ens. 0.060 ± 0.014 0.063 ± 0.012 0.060 ± 0.010 0.054 ± 0.012 0.052 ± 0.013 0.054 ± 0.013 0.054 ± 0.013 0.054 ± 0.011 -
ExtraTrees TabDPT TabM XGBoost RealMLP CatBoost LightGBM TabICLv2 TabPFN-2.6 Metadata Samples 6,819 Features 92 Problem binary Splits 9 Type IID Missingness 0.0% Domain finance Source UCI Year 2009
0.062
Test Error (1-AUC)
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Tuned
0.060 0.058 0.056 0.054 0.052 0.050 0.048
Op tim al
Default
101
102
103
Train time (s)
Figure G.132: taiwanese_bankruptcy_prediction: per-method test error (left) and HPO Pareto trajectory (right).
11.34 ± 1.54 11.48 ± 1.48 11.23 ± 1.60 11.08 ± 1.67 11.08 ± 1.60 11.15 ± 1.68 11.09 ± 1.57 11.15 ± 1.51 -
Dataset: telemonitoring_parkinsons_biomTabPFN-2.6
TabDPT TabICLv2 RandomForest ExtraTrees TabM XGBoost LightGBM CatBoost RealMLP Metadata Samples 502 Features 19 Problem regression Splits 60 Type grouped Missingness 0.0% Domain medical & healthcare Source UCI Year 2007
12.2 12.0 11.8 11.6 11.4 11.2 al
Tuned + Ens.
11.63 ± 1.52 11.46 ± 1.49 11.24 ± 1.60 11.09 ± 1.68 11.10 ± 1.60 11.16 ± 1.68 11.23 ± 1.55 11.14 ± 1.52 -
tim
Tuned
11.94 ± 1.46 11.85 ± 1.53 11.39 ± 1.52 11.08 ± 1.70 11.15 ± 1.64 11.13 ± 1.71 11.17 ± 1.47 11.29 ± 1.42 12.01 ± 1.45 12.30 ± 1.66 11.79 ± 1.52
Test Error (RMSE)
Default
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
102
Train time (s)
103
Figure G.133: telemonitoring_parkinsons_biomedical_voice_measurements: per-method test error (left) and HPO Pareto trajectory (right).
0.170 ± 0.045 0.024 ± 0.025 0.014 ± 0.007 0.016 ± 0.015 0.021 ± 0.015 0.022 ± 0.011 0.016 ± 0.012 0.025 ± 0.017 -
Dataset: thyroid_discordant 0.030 0.025 0.020 0.015
TabM RandomForest XGBoost LightGBM TabDPT RealMLP CatBoost TabPFN-2.6 ExtraTrees TabICLv2 Metadata Samples 3,711 Features 26 Problem binary Splits 30 Type IID Missingness 2.1% Domain medical & healthcare Source UCI Year 1986
al
Tuned + Ens.
0.169 ± 0.043 0.024 ± 0.027 0.014 ± 0.007 0.014 ± 0.010 0.027 ± 0.021 0.024 ± 0.013 0.024 ± 0.019 0.032 ± 0.026 -
0.010
tim
Tuned
0.164 ± 0.044 0.022 ± 0.022 0.021 ± 0.012 0.019 ± 0.012 0.028 ± 0.024 0.022 ± 0.010 0.027 ± 0.025 0.030 ± 0.026 0.018 ± 0.011 0.013 ± 0.007 0.011 ± 0.005
Test Error (1-AUC)
Default
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
101
102
Train time (s)
103
Figure G.134: thyroid_discordant: per-method test error (left) and HPO Pareto trajectory (right). 90
0.163 ± 0.024 0.040 ± 0.011 0.055 ± 0.011 0.042 ± 0.009 0.042 ± 0.010 0.040 ± 0.009 0.042 ± 0.009 0.044 ± 0.011 -
0.163 ± 0.024 0.040 ± 0.010 0.051 ± 0.010 0.041 ± 0.009 0.042 ± 0.010 0.041 ± 0.009 0.042 ± 0.009 0.043 ± 0.010 -
0.0550 0.0525 0.0500 0.0475 0.0450 0.0425 0.0400
ExtraTrees TabM RealMLP CatBoost LightGBM TabPFN-2.6 RandomForest XGBoost TabICLv2 TabDPT Metadata Samples 954 Features 6 Problem binary Splits 30 Type IID Missingness 0.0% Domain business & marketing Source Kaggle Year 2021
al
0.166 ± 0.023 0.051 ± 0.010 0.058 ± 0.011 0.042 ± 0.008 0.041 ± 0.009 0.040 ± 0.009 0.054 ± 0.011 0.047 ± 0.011 0.039 ± 0.010 0.041 ± 0.009 0.040 ± 0.009
Dataset: tour_travels_churn
0.0575
tim
Tuned + Ens.
Op
Tuned
Test Error (1-AUC)
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
101
102
Train time (s)
103
Figure G.135: tour_travels_churn: per-method test error (left) and HPO Pareto trajectory (right).
Tuned + Ens.
0.035
0.031 ± 0.003 0.032 ± 0.005 0.033 ± 0.005 0.019 ± 0.008 0.023 ± 0.005 0.025 ± 0.004 0.010 ± 0.004 0.004 ± 0.001 0.015 ± 0.003 0.014 ± 0.002 0.009 ± 0.002
0.025 ± 0.003 0.034 ± 0.006 0.033 ± 0.005 0.008 ± 0.005 0.002 ± 0.000 0.019 ± 0.004 0.006 ± 0.005 0.003 ± 0.001 -
0.025 ± 0.003 0.033 ± 0.005 0.034 ± 0.005 0.008 ± 0.005 0.002 ± 0.000 0.019 ± 0.004 0.006 ± 0.005 0.003 ± 0.001 -
0.030
Test Error (RMSE)
Tuned
RandomForest XGBoost TabDPT TabPFN-2.6 TabICLv2 CatBoost RealMLP TabM LightGBM Metadata Samples 12,288 Features 38 Problem regression Splits 9 Type grouped Missingness 4.8% Domain technology & internet Source OpenML Year 2020
0.025 0.020 0.015 0.010
tim al
0.005 Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Dataset: video_game_fps_prediction ExtraTrees
Default
101
102
103
Train time (s)
Figure G.136: video_game_fps_prediction: per-method test error (left) and HPO Pareto trajectory (right).
Tuned + Ens.
0.434 ± 0.017 0.236 ± 0.033 0.227 ± 0.036 0.221 ± 0.029 0.226 ± 0.030 0.224 ± 0.031 0.214 ± 0.034 0.221 ± 0.033 -
0.423 ± 0.015 0.236 ± 0.032 0.228 ± 0.036 0.218 ± 0.027 0.222 ± 0.028 0.223 ± 0.029 0.208 ± 0.032 0.217 ± 0.034 -
Dataset: video_transcoding_time_predictTabDPT
TabPFN-2.6 TabICLv2 RandomForest ExtraTrees XGBoost LightGBM CatBoost TabM RealMLP Metadata Samples 68,784 Features 18 Problem regression Splits 9 Type grouped Missingness 0.0% Domain technology & internet Source UCI Year 2015
0.30 0.28 0.26 0.24 0.22 al
Tuned
0.432 ± 0.019 0.248 ± 0.029 0.227 ± 0.036 0.220 ± 0.028 0.224 ± 0.026 0.230 ± 0.028 0.214 ± 0.030 0.222 ± 0.027 0.309 ± 0.023 0.263 ± 0.018 0.246 ± 0.019
Test Error (RMSE)
Default
Op tim
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
101
102
103
Train time (s)
104
Figure G.137: video_transcoding_time_prediction: per-method test error (left) and HPO Pareto trajectory (right).
0.356 ± 0.025 0.261 ± 0.020 0.268 ± 0.022 0.239 ± 0.022 0.257 ± 0.021 0.256 ± 0.020 0.239 ± 0.021 0.238 ± 0.026 -
0.354 ± 0.024 0.259 ± 0.019 0.258 ± 0.019 0.239 ± 0.021 0.257 ± 0.020 0.256 ± 0.020 0.241 ± 0.022 0.238 ± 0.025 -
0.29
Test Error (logloss)
0.356 ± 0.022 0.294 ± 0.025 0.295 ± 0.023 0.255 ± 0.019 0.258 ± 0.020 0.260 ± 0.022 0.259 ± 0.032 0.247 ± 0.026 0.223 ± 0.021 0.221 ± 0.022 0.221 ± 0.023
Dataset: website_phishing
Tuned + Ens. 0.28 0.27 0.26 0.25 0.24 0.23 0.22
tim al
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Tuned
RandomForest ExtraTrees LightGBM XGBoost RealMLP CatBoost TabM TabDPT TabPFN-2.6 TabICLv2 Metadata Samples 1,353 Features 9 Problem multiclass Classes 3 Splits 30 Type IID Missingness 0.0% Domain technology & internet Source UCI Year 2014
Op
Default
101
102
Train time (s)
103
Figure G.138: website_phishing: per-method test error (left) and HPO Pareto trajectory (right). 91
0.160 ± 0.005 0.157 ± 0.002 0.157 ± 0.001 0.127 ± 0.001 0.125 ± 0.001 0.125 ± 0.001 0.123 ± 0.001 0.123 ± 0.001 -
0.159 ± 0.005 0.157 ± 0.001 0.155 ± 0.001 0.127 ± 0.001 0.125 ± 0.001 0.125 ± 0.001 0.122 ± 0.001 0.122 ± 0.001 -
0.18 0.17 0.16 0.15 0.14 0.13
TabDPT RandomForest ExtraTrees TabICLv2 CatBoost TabPFN-2.6 LightGBM XGBoost RealMLP TabM Metadata Samples 127,358 Features 243 Problem binary Splits 9 Type IID Missingness 32.9% Domain medical & healthcare Source Kaggle Year 2021
al
0.160 ± 0.005 0.165 ± 0.001 0.172 ± 0.001 0.128 ± 0.001 0.133 ± 0.001 0.130 ± 0.001 0.126 ± 0.001 0.124 ± 0.001 0.190 ± 0.002 0.126 ± 0.001 0.143 ± 0.002
Dataset: wids_diabetes_mellitus 0.19
Test Error (1-AUC)
Tuned + Ens.
0.12
tim
Tuned
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
102
103
Train time (s)
104
Figure G.139: wids_diabetes_mellitus: per-method test error (left) and HPO Pareto trajectory (right).
0.730 ± 0.006 0.601 ± 0.008 0.597 ± 0.010 0.599 ± 0.009 0.610 ± 0.010 0.605 ± 0.010 0.604 ± 0.007 0.615 ± 0.009 -
0.729 ± 0.006 0.605 ± 0.008 0.598 ± 0.009 0.599 ± 0.009 0.610 ± 0.009 0.606 ± 0.010 0.594 ± 0.008 0.609 ± 0.008 -
LightGBM TabM XGBoost RandomForest CatBoost ExtraTrees RealMLP TabPFN-2.6 TabICLv2 TabDPT Metadata Samples 6,497 Features 12 Problem regression Splits 9 Type IID Missingness 0.0% Domain chemistry & material science Source UCI Year 2009
0.62 0.61 0.60 0.59 al
0.730 ± 0.006 0.619 ± 0.009 0.614 ± 0.009 0.615 ± 0.009 0.623 ± 0.010 0.617 ± 0.008 0.617 ± 0.009 0.629 ± 0.009 0.584 ± 0.008 0.588 ± 0.007 0.587 ± 0.008
Dataset: wine_quality
0.63
Test Error (RMSE)
Tuned + Ens.
tim
Tuned
Op
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
101
102
Train time (s)
103
Figure G.140: wine_quality: per-method test error (left) and HPO Pareto trajectory (right).
0.471 ± 0.018 0.448 ± 0.024 0.435 ± 0.023 0.395 ± 0.024 0.409 ± 0.021 0.405 ± 0.022 0.371 ± 0.023 0.389 ± 0.023 0.386 ± 0.027 0.359 ± 0.019 0.385 ± 0.025
0.424 ± 0.018 0.427 ± 0.023 0.419 ± 0.023 0.393 ± 0.023 0.379 ± 0.023 0.398 ± 0.022 0.365 ± 0.023 0.375 ± 0.023 -
0.417 ± 0.019 0.429 ± 0.023 0.419 ± 0.023 0.393 ± 0.024 0.379 ± 0.023 0.395 ± 0.022 0.363 ± 0.022 0.374 ± 0.023 -
Dataset: wine_world_cost
RandomForest ExtraTrees XGBoost CatBoost TabDPT TabICLv2 LightGBM TabM RealMLP TabPFN-2.6 Metadata Samples 1,279 Features 262 Problem regression Splits 30 Type IID Missingness 10.5% Domain business & marketing Source Kaggle Year 2023
0.44 0.42 0.40 0.38 0.36
al
Tuned + Ens.
Test Error (RMSE)
Tuned
Op tim
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Default
102
104
103
Train time (s)
Figure G.141: wine_world_cost: per-method test error (left) and HPO Pareto trajectory (right).
4.9 ± 0.2 4.0 ± 0.1 4.0 ± 0.1 3.9 ± 0.1 3.9 ± 0.1 3.9 ± 0.1 3.9 ± 0.1 3.9 ± 0.1 -
300 250 200 150 100 50
TabDPT RandomForest ExtraTrees TabICLv2 TabPFN-2.6 XGBoost LightGBM CatBoost TabM RealMLP Metadata Samples 92,629 Features 20 Problem regression Splits 9 Type grouped Missingness 23.5% Domain technology & internet Source HuggingFace Year 2023
al
5.0 ± 0.2 4.0 ± 0.1 4.0 ± 0.1 3.9 ± 0.1 3.9 ± 0.1 3.9 ± 0.1 3.9 ± 0.1 11.1 ± 14.8 -
350
Test Error (RMSE)
5.5 ± 0.2 4.3 ± 0.1 4.2 ± 0.1 3.9 ± 0.1 3.9 ± 0.1 4.0 ± 0.1 3.9 ± 0.1 369.8 ± 1086.7 4.3 ± 0.1 4.0 ± 0.2 4.0 ± 0.1
Dataset: 5g_energy_consumption
Tuned + Ens.
tim
Linear RandomForest ExtraTrees CatBoost LightGBM XGBoost RealMLP TabM TabDPT TabPFN-2.6 TabICLv2
Tuned
0
Op
Default
101
102
103
Train time (s)
104
Figure G.142: 5g_energy_consumption: per-method test error (left) and HPO Pareto trajectory (right).
92
H
NeurIPS Paper Checklist 1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: We claim in the abstract and introduction that we (1) introduce BeyondArena as benchmark and DataFoundry as a unified dataset curation framework and metadata schema, (2) curate 142 datasets spanning diverse disciplines and tabular learning settings, (3) support benchmarking across IID, temporal, and grouped prediction tasks, (4) evaluate models across varying sample sizes, feature dimensionalities, and feature types, and (5) tabular foundation models excel on tiny- to medium-sized IID datasets, and (6) traditional tree-based and deep learning models continue to dominate on non-IID, large-scale, and high-dimensional datasets. We deliver on these claims in Section 4 (1,2); Section 5 (1,3); and Section 6 (4,5,6). Guidelines: • The answer [N/A] means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 2. Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discuss limitations in Section 7. Guidelines: • The answer [N/A] means that the paper has no limitation while the answer [No] means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate “Limitations” section in their paper. • The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best 93
judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 3. Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [N/A] Justification: We present no theoretical results. Guidelines: • The answer [N/A] means that the paper does not include theoretical results. • All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems. • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 4. Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: We describe our dataset curation in Section 4, the experimental setup in Section 5, provide more details in the Appendix C.2, and share all our code. Guidelines: • The answer [N/A] means that the paper does not include experiments. • If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). 94
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 5. Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We share all our dataset curation code, benchmarking code, and dataset artifacts in the paper. Guidelines: • The answer [N/A] means that paper does not include experiments requiring code. • Please see the NeurIPS code and data submission guidelines (https://neurips.cc/ public/guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //neurips.cc/public/guides/CodeSubmissionPolicy) for more details. • The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc. • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 6. Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results? Answer: [Yes] Justification: We share all details in Section 5. Guidelines: • The answer [N/A] means that the paper does not include experiments. • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 7. Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: We report all results with error bars and provide a statistical analysis in Appendix E. Guidelines: • The answer [N/A] means that the paper does not include experiments. 95
• The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates). • If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 8. Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: We provide details on the compute resources in Section 5 and Appendix C.2. Guidelines: • The answer [N/A] means that the paper does not include experiments. • The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper). 9. Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: We believe our work conforms to the NeurIPS Code of Ethics. Guidelines: • The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics. • If the authors answer [No], they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We discuss societal impacts in Section 7. 96
Guidelines: • The answer [N/A] means that there is no societal impact of the work performed. • If the authors answer [N/A] or [No], they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 11. Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)? Answer: [N/A] Justification: The code and curated data artifacts we release have no perceivable risk of misuse. We do not release models. Guidelines: • The answer [N/A] means that the paper poses no such risks. • Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We credited all datasets we curated, see Table B.1, and respected the licenses of all assets (e.g., code and models) that we used. Guidelines: • The answer [N/A] means that the paper does not use existing assets. • The authors should cite the original paper that produced the code package or dataset. • The authors should state which version of the asset is used and, if possible, include a URL. 97
• The name of the license (e.g., CC-BY 4.0) should be included for each asset. • For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset’s creators. 13. New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: We introduce DataFoundry in Section 4 and share its documented code. Guidelines: • The answer [N/A] means that the paper does not release new assets. • Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 14. Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [N/A] Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines: • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 15. Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [N/A] Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines: • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects. 98
• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 16. Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required. Answer: [N/A] Justification: Our contributions do not involve LLMs in any components. Guidelines: • The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components. • Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.
99