arXiv:2605.18696v1 [cs.LG] 18 May 2026
Aditya Tanna, Yash Desai, Pratinav Seth, Mohamed Bouadi, Nassim Bouarour, Vinay Kumar Sankarapu Lexsi Labs Abstract Tabular foundation models (TFMs) now match or beat tuned gradient-boosted trees on a growing fraction of tabular tasks, but no single TFM wins on every dataset. Ensembling is the go to fix here, and it works less well than expected. Six modern TFMs form a near-redundant pool: their mean pairwise Q-statistic is 0.961, close enough to 1 that any convex combination is bounded above. We benchmark six ensemble strategies over six TFMs on 153 OpenML classification tasks. The best ensemble, two-level cascade stacking, buys +0.18% accuracy over the strongest single TFM at 253× the compute. A Friedman and Nemenyi analysis places three ensembles and the best base TFM in a single equivalence group; three other ensembles are significantly worse than the best base. Stacking with a logistic-regression meta-learner is the most striking case: competitive accuracy and ROC-AUC, the worst log-loss rank among the ensembles. The meta-learner improves accuracy by sharpening class boundaries, which destroys calibration. We recommend greedy selection as the practical default. Keywords: Tabular foundation models, ensembling, benchmarking, calibration, diversity
1
Introduction
Tabular foundation models (TFMs) have advanced rapidly, with growing architectural diversity. We show that this diversity is mostly nominal: a pool of six modern TFMs produces near-redundant predictions, and ensembling cannot exploit diversity that is not there. TabPFN [1, 2], TabICL [3], and variants like Mitra [4], Orion-Bix [5], TabDPT [6], and CARTE [7] all perform in-context learning (ICL) over synthetic or curated priors. Continued pre-training on real-world tables [8] now rivals heavy AutoML in a single forward pass: TabPFNv2.5 matches AutoGluon 1.4 on TabArena [9]. What these models do not do, however, is win uniformly. Across our 153-dataset benchmark, the per-dataset accuracy wins among six TFMs split as follows: TabPFNv2.5-52, TabICLv2-46, TabPFNv2.6-25, TabICL-13, LimiX-12, OrionMSPv1.5-5 (all approximate numbers). The median accuracy spread between best and worst TFM on a single dataset is 1.95%; on 24% of datasets it exceeds 5%, so committing to one TFM in advance loses meaningful accuracy on roughly a quarter of the benchmark. Ensembling is the textbook response [10, 11, 12]. The cost-benefit story for TFMs differs from the gradient-boosted decision tree (GBDT) setting where ensemble methods originated. Per-task TFM inference is cheap once the model is pretrained (under one second on most benchmark datasets), though pretraining itself is a substantial fixed cost. Cascade-style stacking layers K-fold OOF inference across all bases, dwarfing any single forward pass. Beyond compute, there is a structural concern. TFMs trained with ICL on synthetic priors approximate Bayesian model averaging at inference [13, 14]; broadly similar priors yield broadly similar posteriors, leaving little for a convex combiner to recover.
Ensembling Tabular Foundation Models
Position relative to existing work. Recent work ensembles TFMs differently. The post-hoc ensemble protocol bundled with TabPFN [2] ensembles hyperparameter configurations of one TFM; TabICLv2 [3] does the same internally with column and class shuffles. TabArena [9] measures post-hoc ensembling across heterogeneous model classes (TFMs, GBDTs, neural baselines) and reports that some classes are over-represented in cross-class ensembles due to validation overfitting. HAPEns [15] adds hardware-aware multi-objective selection on a general model pool. TabM [16] pursues parameter-efficient ensembling at the architecture level. Recent architectural work in the same TFM family includes Orion-BiX [5], while [17] study fine-tuning protocols (zero-shot, meta-learning, SFT, PEFT) across the same CC18/TALENT/TabZilla benchmarks we use here. We isolate a different question: holding the model class fixed, what does ensembling six different TFMs buy on its own? Section 2 situates the answer against the broader ensemble and tabular-deep-learning literature. Contributions. 1. A diversity-ceiling diagnostic. Six modern TFMs share an ICL-on-synthetic-priors recipe and produce near-redundant predictions (Q = 0.961), bounding the gain available to any convex combiner. 2. An empirical accuracy/compute frontier with a calibration overlay across six ensemble strategies on 153 OpenML tasks. The best ensemble buys a sub-percent accuracy gain at 253× the compute of the strongest base; three of six ensembles are significantly worse than the best base. 3. Calibration findings. Stacking with a logistic-regression meta-learner has competitive accuracy and ROC-AUC ranks but the worst log-loss rank among the ensembles; the meta-learner sharpens class boundaries, improving accuracy at the cost of probability quality.
2
Related Work
The ensemble literature provides a useful baseline for what to expect from this study. Classical results show that gains from averaging classifiers scale with two things: how accurate the base classifiers are, and how independent their errors are. Bagging [10] reduces variance when bases are decorrelated; stacking [18, 19] learns a meta-mapping over base outputs; ensemble selection from a large library of fits [12] chooses a small weighted subset by greedy validation search. Across all three, the standard assumption is that the pool was constructed to be diverse, often by training the same model class on resampled or perturbed data. Diversity measures such as the Q-statistic, Cohen’s κ, and disagreement [20] were introduced to quantify exactly that. Tabular foundation models break the assumption. The bases are pretrained, not refit per dataset, and the per-task perturbation budget is small: random seeds change the in-context order but not the underlying prior. Recent work has explored ensembling in this setting at three different levels. At the architecture level, TabM [16] shares parameters across an internal set of branches and trains them end-to-end. At the configuration level, the post-hoc protocol in TabPFN [2] averages many hyperparameter configurations of a single model. At the cross-class level, TabArena [9] reports post-hoc ensembles over TFMs, GBDTs, and neural baselines, and notes that validation-based weight selection can over-represent some model classes. HAPEns [15] extends that idea with a hardware-aware multi-objective selector. We sit at a fourth level: holding the model class fixed (all TFMs), and asking what convex or stacked combinations of six different pretrained TFMs can recover on their own. The notion that ensemble gains have a ceiling is not new [20, 11], but it usually appears as a property of small classifier pools on small datasets, not as a structural feature of a pretraining family. For TFMs, the question becomes whether the inductive bias of ICL over synthetic priors leaves enough room for diversity to matter. There is theoretical reason to expect that it does not: [13] and [14] formalise the sense in which ICL approximates Bayesian model averaging at inference time, so two TFMs trained on broadly similar priors are already implicitly averaging over broadly similar posterior families. The Q-statistic we report (Q̄ = 0.961) is the empirical counterpart of that argument: six models that all consult the same kind of prior fail on the same instances. Real-TabPFN [8] suggests one possible escape route, namely continued pre-training on real-world tables to shift the prior; whether that produces enough error decorrelation to lift the ceiling is open. Calibration is the second thread. Modern deep classifiers tend to be overconfident, and post-hoc fixes such as temperature scaling [21] were developed specifically for that regime. [22] showed that some ensemble methods (notably bagged trees and random forests) produce well-calibrated probabilities almost as a byproduct, while others (boosting) do not. The pattern we find in §5.3 sits in the same family of results: convex combiners of TFM probabilities preserve the calibration of their inputs, but a discriminative meta-learner trained on out-of-fold predictions does not. Selective-prediction and worst-group metrics [23, 24] let us tell the two failure modes apart, which is why we report them alongside accuracy. 2
Ensembling Tabular Foundation Models
3
Ensemble Strategies
Six strategies share a common fit/predict/predict_proba interface over a fixed pool of K base TFMs producing class-probability vectors pk (x). P Weighted Averaging (WA) p̂ = k wk pk with wk ∝ scorek on validation. No second stage. Cheapest combiner. Greedy Selection [12]. Forward selection with replacement: at each of S iterations, the base whose addition maximises validation accuracy is added. Final weight equals selection count over S. We use S = 50, matching the AutoGluon [25] WeightedEnsembleModel default. Stacking [18, 19]. Bases produce 5-fold out-of-fold (OOF) predictions. A logistic-regression meta-learner is trained on the OOF features. Temperature-Scaled Blending [21]. Per-base temperature Tk is fit on the validation set by minimising negative log-likelihood (NLL) of softmax(log pk /Tk ); calibrated probabilities are then averaged uniformly. Cascade Stacking. Two-level stacking with skip connections, modifying AutoGluon’s high-quality preset [25]. Level-1 OOF predictions concatenate with raw features and feed level-2 base models, also with K-fold OOF. A final greedy-selection layer combines all level outputs. We use 2 levels, 3-fold OOF, S = 50. Random-Init (Deep) Ensemble [26]. Each TFM is run with M = 3 different seeds. Per-base predictions are averaged across seeds, then cross-base averaging uses performance weights. Strategies are implemented in Python on top of the public TabTune library [27].
4
Experimental Setup
Datasets. 153 OpenML classification tasks drawn from the CC18 [28], TALENT [29], and TabZilla [30] pools. Selection criteria and the full dataset inventory are in Appendix H. Base TFMs. Six models in inference mode: TabPFNv2.5 [31], TabPFNv2.6 [2], TabICL [3], TabICLv2 [3], LimiX [32], OrionMSPv1.5 [33]. Protocol. Per dataset: an 80/20 stratified train/test split; within train, a 75/25 train/validation split for ensemble weight learning. Stacking and cascade levels use 5-fold and 3-fold internal CV respectively. A fixed seed controls splits and base-model initialisation. Metrics. Accuracy, weighted F1, one-vs-rest ROC-AUC, multi-class log-loss, and total fit-time per dataset (seconds). For deeper analysis on the TabArena classification suite we additionally report expected calibration error (ECE) [21], the reliability component of the Brier decomposition [34], area under the risk-coverage curve (AURC) [23], coverage at 95% accuracy, and worst-group accuracy (WGA) [24]. Statistical significance is reported via Friedman [35], Nemenyi [36], and pairwise Wilcoxon signed-rank [37]. Hardware. A single H100 (80 GB) GPU per run.
5
Results
5.1
Aggregate performance
Table 1 reports per-method statistics across the 153 datasets in our benchmark. The accuracy spread among the top eight methods is 0.45 percentage points (TabICL at 0.872 to Cascade at 0.882). The Friedman test rejects equality of mean ranks across the 12 methods (χ2 = 389.95, p < 10−30 ); methods are not exchangeable, but the question is which differences survive a per-pair correction. Calibration, selective-prediction, and group-robustness metrics on the TabArena suite are reported in Table 3 (Appendix C) and analysed in §5.3. The Nemenyi critical difference at α = 0.05 for K = 12, N = 153 is CD = 1.347. Three methods sit within CD of the top-ranked Cascade_2level: Stacking_LR (∆ = 0.48), Greedy_Selection (∆ = 0.80), and TabICLv2 (∆ = 0.90). Three ensembles and one base TFM are statistically indistinguishable on accuracy across 153 tasks; the remaining three ensembles cannot beat the best base. Pairwise Wilcoxon tests sharpen the picture: against TabICLv2, only Cascade_2level wins (+0.18%, p = 0.008); Greedy_Selection (+0.01%) and Stacking_LR (−0.03%) tie; WA, Temp_Scaled, and DeepEnsemble_3seed are all significantly worse (p < 0.05). One ensemble of six clears the bar of beating the strongest base. 3
Ensembling Tabular Foundation Models
Table 1 Mean performance over 153 OpenML classification tasks. Lower rank is better. Methods are sorted by accuracy rank; ensemble methods are italicized. Best in each rank column is bold. Method
Acc.
Acc. rank
Log-loss
Log-loss rank
ROC-AUC
ROC-AUC rank
Fit (s)
Cascade_2level Stacking_LR Greedy_Selection TabICLv2 DeepEnsemble_3seed WA_performance Temp_Scaled TabPFNv2.5 TabPFNv2.6 LimiX TabICL OrionMSPv1.5
0.882 0.880 0.881 0.881 0.879 0.879 0.879 0.878 0.876 0.875 0.872 0.848
4.48 4.96 5.28 5.39 6.10 6.18 6.41 6.44 6.78 7.67 7.96 10.34
0.289 0.288 0.278 0.274 0.281 0.282 0.283 0.286 0.283 0.288 0.289 0.356
4.55 8.13 4.54 4.11 5.62 5.50 6.47 6.64 6.26 7.17 7.61 11.41
0.907 0.907 0.906 0.907 0.907 0.907 0.907 0.904 0.903 0.902 0.900 0.880
5.01 4.59 5.51 4.73 5.58 5.49 6.14 7.69 6.89 7.64 8.02 10.72
178.5 6.6 6.7 0.7 75.7 6.6 6.6 1.2 1.6 0.5 1.0 1.8
5.2
Accuracy/compute frontier
Fit times span four orders of magnitude. TabICLv2 averages 0.71 s per dataset; Greedy, Stacking_LR, WA, and Temp_Scaled all sit near 6.6 s, which is roughly the cost of one forward pass through the six bases plus a thin combination layer. DeepEnsemble_3seed costs 75.7 s, and Cascade_2level costs 178.5 s. Figure 1 plots the trade-off. The Pareto frontier is dominated by TabICLv2 (cheapest competitive option) and Greedy_Selection (best accuracy at moderate cost). Cascade_2level sits on the frontier, but its marginal accuracy advantage over TabICLv2 corresponds to a 253× compute multiplier. DeepEnsemble_3seed is dominated outright: WA_performance and Stacking_LR achieve similar or better accuracy at one tenth its cost. Figure 2 (Appendix B) shows the corresponding critical-difference diagram.
5.3
Calibration and the diversity ceiling
Log-loss diverges from accuracy. The log-loss column of Table 1 tells a different story than the accuracy column. TabICLv2 has the lowest log-loss rank (4.11); Greedy_Selection (4.54) and Cascade_2level (4.55) sit close behind, both producing convex combinations of probability vectors. Stacking_LR ranks 8.13, the worst of any method tested except OrionMSPv1.5. Linear stacking still places the right class label, which is why its accuracy and ROC-AUC ranks stay competitive, but the cross-entropy objective on OOF predictions pushes the meta-learner toward sharper probability outputs than the bases produce, which degrades calibration. Calibration tracks combination strategy, not compute. Table 3 (Appendix C) reports five complementary metrics on the TabArena classification suite: ECE, Brier reliability, AURC, coverage at 95% accuracy, and worstgroup accuracy. TabICLv2 sets the calibration ceiling (ECE = 0.0236, Brier-REL = 0.0024). Greedy_Selection is the only ensemble that approaches it (ECE = 0.0253), and it never optimises for calibration directly. Stacking_LR records the worst calibration of any ensemble (ECE = 0.0272, Brier-REL = 0.0031), consistent with its log-loss rank. Temperature-Scaled Blending is equally instructive: per-base NLL minimisation gives an ECE of 0.0273, no better than Stacking_LR’s. Base-model diversity caps uncertainty quality. The mean pairwise Q-statistic [20] across the six TFMs is 0.961 (σ = 0.183, Cohen’s κ = 0.856, in the “almost perfect agreement” band of the conventional Landis-Koch scale). Q values close to 1 signal near-redundancy: the six bases share the ICL-on-synthetic-priors recipe and tend to fail on the same instances, so any convex combiner has little variance to absorb; Appendix D states this ceiling formally as a consensus-set bound on the ensemble-vs-base accuracy gap. The ceiling is most consequential for DeepEnsemble_3seed: three random seeds perturb context order but share the synthetic prior, producing an AURC of 0.0617 (28% above TabICLv2’s 0.0483) and coverage at 95% accuracy of 62.1% versus 68.4%. The 75.7 s cost buys neither accuracy nor uncertainty improvement; for selective prediction, Greedy_Selection (AURC = 0.0484) is the appropriate choice. Cascade earns its cost on group robustness. Worst-group accuracy is the one axis where heavy stacking earns its overhead. Cascade_2level reaches 0.803, on par with TabICLv2 (0.802) and outperforming all other ensembles by roughly three points (Greedy and Stacking_LR both at 0.776). The skip-connection architecture appears to implicitly down-weight base models that are systematically biased on minority subgroups; simpler convex combiners do not 4
Ensembling Tabular Foundation Models
Figure 1 Mean accuracy vs. mean fit time (log scale, seconds per dataset) over 153 OpenML tasks. Pareto-optimal methods are marked. Cascade buys an accuracy advantage over TabICLv2 for 253× the compute.
replicate this. The fairness margin is narrow and confined to sensitive-attribute datasets, but it is the one regime in which cascade’s 253× compute overhead translates into a qualitative advantage rather than a marginal one. Per-dataset patterns. Aggregate means hide per-dataset behaviour. Cascade_2level beats the per-task oracle base on 42 of 153 tasks (ties on 36, loses on 75, mean ∆ = −0.0025); against TabICLv2 specifically, 70/41/42 (mean ∆ = +0.0018). Ensembles win on average not because they are uniformly better, but because no fixed base is best on every dataset. The Pearson correlation between ensemble gain and inter-base accuracy spread is −0.03, so the ceiling is roughly uniform across dataset shapes; ensembling helps no more on high-spread datasets than on low-spread ones.
6
Discussion
The shape of the result holds across every metric we report. TFMs trained with ICL on synthetic priors already approximate Bayesian model averaging at inference, so explicit downstream ensembling lands inside the noise floor of a strong single base. Cascade stacking buys the last 0.2% of accuracy by letting the meta-learner combine raw features alongside OOF predictions, but the cost-benefit ratio is poor outside competition settings. Greedy selection is the more honest default: roughly 10× the cost of the strongest single model, the same mean accuracy as the heaviest stack, and no calibration regression. Two patterns matter for downstream work. First, calibration is not a free byproduct of accuracy ensembling. Stacking with logistic regression damages probability quality despite improving accuracy rank, and uniform averaging after per-base temperature scaling is no better than the best base. Calibration-aware meta-learners optimised for log-loss directly, or post-hoc recalibration applied to the ensemble output rather than to its members, both remain open. Second, the per-dataset variance in best-base identity is what ensembling is really being asked to solve. A small gating learner trained on dataset metafeatures to pick the best TFM per dataset might match cascade at far lower compute, and is a better fit to the structure of the problem than a stack.
6.1
Limitations.
We use a single seed per dataset; statistical power comes from across-dataset variation rather than within-dataset replicates, which is what paired Wilcoxon, Friedman, and Nemenyi tests assume on N = 153 tasks. A small number of the largest tasks were dropped for individual base TFMs due to memory constraints, and Table 1 reports the intersection of tasks that completed for every method. We do not include GBDT baselines; the comparison is between single TFMs and TFM-only ensembles, leaving open whether out-of-class diversity (TFM + GBDT) recovers gains the within-class pool cannot. 5
Ensembling Tabular Foundation Models
6.2
Contamination.
Several base TFMs were pretrained on data overlapping with OpenML, a concern raised by the TabArena protocol [9]. Reported deltas should be read as upper bounds on within-pool ensemble effects: under cleaner contamination protocols, the ensemble-vs-best-base gap is likely smaller, not larger. Contamination does not change the qualitative finding (a near-redundant pool produces a hard ceiling), but the precise accuracy delta is best treated as inflated. Future work. Hybrid TFM+GBDT pools, per-dataset gating learners, time-series TFMs where base-model spread may be larger, and calibration-aware meta-learners.
7
Conclusion
Six tabular foundation models trained with ICL on synthetic priors form a near-redundant pool. A Q-statistic of 0.961 caps what any convex combiner can recover, and the empirical results follow: a top equivalence group of four methods (three ensembles plus the best base) statistically indistinguishable on accuracy; a best ensemble that recovers a sub-percent accuracy gain at 253× compute; and a calibration trap when a meta-learner is asked to manufacture extra accuracy by sharpening probabilities. Greedy selection is the practical default; cascade stacking is justifiable only when worst-group accuracy is a primary target. The open question is whether out-of-class diversity (TFM + GBDT) breaks through the ceiling that within-class pools cannot.
References [1] Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. TabPFN: A transformer that solves small tabular classification problems in a second. In International Conference on Learning Representations (ICLR), 2023. [2] Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirrmeister, and Frank Hutter. Accurate predictions on small data with a tabular foundation model. Nature, 637:319–326, 2025. [3] Jingang Qu, David Holzmüller, Gaël Varoquaux, and Marine Le Morvan. TabICL: A tabular foundation model for in-context learning on large data. In International Conference on Machine Learning (ICML), 2025. [4] Xiyuan Zhang, Danielle C. Maddix, Junming Yin, Nick Erickson, Abdul Fatir Ansari, Boran Han, Shuai Zhang, Leman Akoglu, Christos Faloutsos, Michael W. Mahoney, Cuixiong Hu, Huzefa Rangwala, George Karypis, and Bernie Wang. Mitra: Mixed synthetic priors for enhancing tabular foundation models. arXiv preprint arXiv:2510.21204, 2025. [5] Mohamed Bouadi, Pratinav Seth, Aditya Tanna, and Vinay Kumar Sankarapu. Orion-Bix: Bi-Axial attention for tabular in-context learning. In Proceedings of the ACM Web Conference 2026, WWW ’26, New York, NY, USA, 2026. Association for Computing Machinery. [6] Junwei Ma, Valentin Thomas, Rasa Hosseinzadeh, Hamidreza Kamkari, Alex Labach, Jesse C Cresswell, Keyvan Golestan, Guangwei Yu, Maksims Volkovs, and Anthony L Caterini. TabDPT: Scaling tabular foundation models. arXiv preprint arXiv:2410.18164, 2024. [7] Myung Jun Kim, Léo Grinsztajn, and Gaël Varoquaux. Carte: Pretraining and transfer for tabular learning, 2024. [8] Anurag Garg, Muhammad Ali, Noah Hollmann, Lennart Purucker, Samuel Müller, and Frank Hutter. RealTabPFN: Improving tabular foundation models via continued pre-training with real-world data. arXiv preprint arXiv:2507.03971, 2025. [9] Nick Erickson, Lennart Purucker, Andrej Tschalzev, David Holzmüller, Prateek Mutalik Desai, David Salinas, and Frank Hutter. TabArena: A living benchmark for machine learning on tabular data. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2025. [10] Leo Breiman. Bagging predictors. Machine Learning, 24:123–140, 1996. [11] Thomas G. Dietterich. Ensemble methods in machine learning. In Multiple Classifier Systems, volume 1857 of Lecture Notes in Computer Science, pages 1–15. Springer, 2000. [12] Rich Caruana, Alexandru Niculescu-Mizil, Geoff Crew, and Alex Ksikes. Ensemble selection from libraries of models. In International Conference on Machine Learning (ICML), 2004. [13] Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. Transformers can do bayesian inference. In International Conference on Learning Representations, 2022. [14] Yufeng Zhang, Fengzhuo Zhang, Zhuoran Yang, and Zhaoran Wang. What and how does in-context learning learn? bayesian model averaging, parameterization, and generalization. In Yingzhen Li, Stephan Mandt, Shipra 6
Ensembling Tabular Foundation Models
Agrawal, and Emtiyaz Khan, editors, Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, volume 258 of Proceedings of Machine Learning Research, pages 1684–1692. PMLR, 3–5 May 2025. [15] Jannis Maier and Lennart Purucker. HAPEns: Hardware-aware post-hoc ensembling for tabular data. arXiv preprint arXiv:2603.10582, 2026. [16] Yury Gorishniy, Akim Kotelnikov, and Artem Babenko. TabM: Advancing tabular deep learning with parameterefficient ensembling. In International Conference on Learning Representations (ICLR), 2025. [17] Aditya Tanna, Pratinav Seth, Mohamed Bouadi, and Vinay Kumar Sankarapu. Exploring fine-tuning for tabular foundation models. In Proceedings of the ACM Web Conference 2026, WWW ’26, New York, NY, USA, 2026. Association for Computing Machinery. [18] David H. Wolpert. Stacked generalization. Neural Networks, 5(2):241–259, 1992. [19] Kai Ming Ting and Ian H. Witten. Issues in stacked generalization. Journal of Artificial Intelligence Research, 10:271–289, 1999. [20] Ludmila I. Kuncheva and Christopher J. Whitaker. Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Machine Learning, 51(2):181–207, 2003. [21] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning (ICML), 2017. [22] Alexandru Niculescu-Mizil and Rich Caruana. Predicting good probabilities with supervised learning. In Proceedings of the 22nd International Conference on Machine Learning, volume 119 of ACM International Conference Proceeding Series, pages 625–632. ACM, 2005. [23] Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2017. [24] Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In International Conference on Learning Representations (ICLR), 2020. [25] Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, and Alexander Smola. AutoGluon-Tabular: Robust and accurate AutoML for structured data. arXiv preprint arXiv:2003.06505, 2020. [26] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems (NeurIPS), 2017. [27] Aditya Tanna, Pratinav Seth, Mohamed Bouadi, Utsav Avaiya, and Vinay Kumar Sankarapu. TabTune: A unified library for inference and fine-tuning tabular foundation models. arXiv preprint arXiv:2511.02802, 2025. [28] Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer, Pieter Gijsbers, Frank Hutter, Michel Lang, Rafael G. Mantovani, Jan N. van Rijn, and Joaquin Vanschoren. OpenML benchmarking suites. In Advances in Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks, 2021. [29] Si-Yang Liu, Hao-Run Cai, Qi-Le Zhou, Huai-Hong Yin, Tao Zhou, Jun-Peng Jiang, and Han-Jia Ye. Talent: A tabular analytics and learning toolbox. Journal of Machine Learning Research, 26(226):1–16, 2025. [30] Duncan McElfresh, Sujay Khandagale, Jonathan Valverde, Vishak Prasad C, Benjamin Feuer, Chinmay Hegde, Ganesh Ramakrishnan, Micah Goldblum, and Colin White. When do neural nets outperform boosted trees on tabular data?, 2023. [31] L’eo Grinsztajn, Klemens Floge, Oscar Key, Felix Birkel, Philipp Jund, Brendan Roof, Benjamin Jager, Dominik Safaric, Simone Alessi, Adrian Hayler, Mihir Manium, Rose Yu, Felix Jablonski, Shi Bin Hoo, Anurag Garg, Jake Robertson, Magnus Buhler, Vladyslav Moroshan, Lennart Purucker, Clara Cornu, Lilly Charlotte Wehrhahn, Alessandro Bonetto, Bernhard Scholkopf, Sauraj Gambhir, Noah Hollmann, and Frank Hutter. Tabpfn-2.5: Advancing the state of the art in tabular foundation models. ArXiv, abs/2511.08667, 2025. [32] Xingxuan Zhang et al. LimiX: Unleashing structured-data modeling capability for generalist intelligence. arXiv preprint arXiv:2509.03505, 2025. [33] Mohamed Bouadi, Pratinav Seth, Aditya Tanna, and Vinay Kumar Sankarapu. Orion-MSP: Multi-scale sparse attention for tabular in-context learning. arXiv preprint arXiv:2511.02818, 2025. [34] Glenn W. Brier. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1):1–3, 1950. [35] Milton Friedman. A comparison of alternative tests of significance for the problem of m rankings. The Annals of Mathematical Statistics, 11(1):86–92, 1940. [36] Peter Nemenyi. Distribution-Free Multiple Comparisons. PhD thesis, Princeton University, 1963. [37] Frank Wilcoxon. Individual comparisons by ranking methods. Biometrics Bulletin, 1(6):80–83, 1945.
7
Ensembling Tabular Foundation Models
A
Method-name glossary
The body and Tables 1 and 3 use compact short-form labels; Figures 3 and 4 render the same methods in long form. Table 2 reconciles the two. Table 2 Short-form labels used in prose and Tables 1 and 3 mapped to the long-form labels rendered in Figures 3 and 4. Short form (prose, tables)
Long form (figures)
Cascade_2level Stacking_LR Greedy_Selection WA_performance Temp_Scaled DeepEnsemble_3seed
Cascade 2-Level Stacking Logistic Regression Stacking Greedy Selection Weighted Averaging Temperature Scaling Deep Ensemble
8
Ensembling Tabular Foundation Models
B
Critical-difference diagram
Figure 2 visualises the rank-difference structure underlying the Friedman and Nemenyi analysis in §5. Methods are positioned along the mean-rank axis; horizontal bars connect groups whose pairwise differences fall within the critical difference CD = 1.347 (so members of a group are statistically indistinguishable on accuracy at α = 0.05).
Figure 2 Critical-difference diagram for accuracy ranks (Friedman + Nemenyi, α = 0.05, CD = 1.347). Methods connected by horizontal bars are statistically indistinguishable. The top equivalence group is Cascade_2level, Stacking_LR, Greedy_Selection, and TabICLv2.
9
Ensembling Tabular Foundation Models
C
Calibration, uncertainty, and group-robustness metrics
Table 3 Calibration, uncertainty, and group-robustness metrics on the TabArena classification suite. ECE: expected calibration error (15-bin). Brier-REL: reliability component of the Brier decomposition. AURC: area under the risk-coverage curve (max-probability scorer). Cov@95: fraction of samples predictable at ≥ 95% accuracy. WGA: worst-group accuracy on sensitive-attribute datasets. Best ensemble and best base are bolded separately. Method
ECE↓
Brier-REL↓
AURC↓
Cov@95↑
WGA↑
Cascade_2level Greedy_Selection Stacking_LR Temp_Scaled WA_performance DeepEnsemble_3seed
0.0261 0.0253 0.0272 0.0273 0.0277 0.0275
0.0025 0.0027 0.0031 0.0030 0.0030 0.0030
0.0485 0.0484 0.0491 0.0485 0.0485 0.0617
0.677 0.679 0.684 0.679 0.679 0.621
0.803 0.776 0.776 0.790 0.790 0.790
TabICLv2 TabICL TabPFNv2.6 TabPFNv2.5 LimiX OrionMSPv1.5
0.0236 0.0261 0.0255 0.0258 0.0252 0.0419
0.0024 0.0028 0.0028 0.0026 0.0027 0.0044
0.0483 0.0498 0.0485 0.0490 0.0496 0.0603
0.684 0.673 0.680 0.670 0.673 0.629
0.802 0.775 0.773 0.779 0.789 0.758
10
Ensembling Tabular Foundation Models
D
Convex-combination ceiling: a formal statement
Proposition (Convex-combination ceiling). Let f1 , . . . , fK be classifiers producing class-probability vectors pk (x) with hard predictions ŷk (x) =P arg maxc pk (x)c , and let C = {x : ŷ (x) = · · · = ŷ (x)} denote the consensus set. For any convex combiner p̂(x) = 1 K k wk pk (x) with P wk ≥ 0 and k wk = 1, the ensemble prediction arg maxc p̂(x)c equals the unanimous label on C. Consequently the accuracy gap between the ensemble and any single base classifier is bounded above by the disagreement-set fraction 1 − |C|/|X |. P Proof sketch. On x ∈ C every pk (x) assigns its maximum to the same label y ∗ , so p̂(x)y∗ = k wk pk (x)y∗ ≥ P ∗ k wk pk (x)c for every other class c, with equality only if every base ties at x. Hence ŷ(x) = y , matching every base on C. Outside C the ensemble can differ from a base on at most |X \ C| inputs, bounding the absolute accuracy gap.□ A high mean pairwise Q-statistic is a sufficient indicator that, when individual error rates are similar across the pool, the disagreement set is small [20]: errors concentrate on the same instances, so the consensus set is large and the bound above tightens. For our six-TFM pool the mean pairwise Q-statistic is Q̄ = 0.961 (computation in Appendix E), close to 1 and signalling a hard diversity ceiling any combiner must work against.
E
Q-statistic computation
The pairwise Q-statistic [20] is computed from each pair of base TFMs’ test-set predictions on each of the 153 tasks. For a pair (fk , fl ) on a task, we form the 2 × 2 contingency table on test instances by (correct, wrong) under each classifier and define ad − bc , Q(task) = kl ad + bc where a, b, c, d are the counts of (both correct), (k correct, l wrong), (k wrong, l correct), and (both wrong) respectively. Per-pair Qkl is the unweightedmean of Q(task) across the 153 tasks. The pool-level Q̄ = 0.961 reported in §5.3 is the kl unweighted mean across the 62 = 15 unordered base-pair indices; σ = 0.183 is the standard deviation across the same set. Cohen’s κ uses the same contingency tables and is reported using the standard chance-adjusted-agreement formula.
11
Ensembling Tabular Foundation Models
F
Head-to-head ensemble comparison
Figure 3 reports per-pair win rates across the 153 tasks for the six ensemble methods. Cascade 2-Level Stacking is the dominant strategy: it wins on 55.6%, 56.2%, and 55.6% of datasets against Weighted Averaging, Temperature Scaling, and Deep Ensemble respectively, and ties or loses only against Logistic Regression Stacking and Greedy Selection. Weighted Averaging never exceeds a 26.8% win rate against any single opponent and Temperature Scaling never exceeds 24.2%, consistent with their ranking in Table 1.
Figure 3 Head-to-head win matrix across 153 OpenML classification tasks (accuracy). Each cell (i, j) reports the percentage of datasets on which row method i outperforms column method j; the diagonal is zero by construction. Cascade 2-Level Stacking is the dominant strategy (55.6%, 56.2%, and 55.6% of datasets against Weighted Averaging, Temperature Scaling, and Deep Ensemble respectively). Logistic Regression Stacking ranks second. Weighted Averaging wins on at most 26.8% of datasets against any single opponent; Temperature Scaling never exceeds 24.2%.
12
Ensembling Tabular Foundation Models
G
Mean rank leaderboard
Figure 4 plots the per-method mean accuracy rank across the 153 tasks, colour-coded by method type (ensemble strategies in orange, base TFMs in blue). Cascade 2-Level Stacking holds the best mean rank (4.48), followed by Logistic Regression Stacking (4.96), Greedy Selection (5.28), and the strongest base TabICLv2 (5.39). The Friedman test rejects equality of mean ranks across the 12 methods at χ2 = 389.95, p ≈ 8.31 × 10−77 ; which pairwise differences survive the per-pair correction is shown in Figure 2.
Figure 4 Mean rank leaderboard for the 12 methods evaluated across 153 datasets. Ensemble strategies (orange) cluster with the strongest base models (blue) at the top of the ranking; Cascade 2-Level Stacking achieves the best rank (4.48), followed by Logistic Regression Stacking (4.96) and Greedy Selection (5.28). Friedman χ2 = 389.95, p ≈ 8.31 × 10−77 .
13
Ensembling Tabular Foundation Models
H
Dataset details
This section lists the full inventory of OpenML benchmark datasets used in our study. Table 4 reports the OpenML identifier, dataset name, sample count, feature count, number of target classes, and source benchmark suite for each task. Table 4 Full inventory of the 153 OpenML benchmark datasets used in this study. ID: OpenML dataset identifier. Samples: number of instances (range: 128 to 581,012; median: 3,196). Feat.: number of input features (range: 5 to 1,777; median: 22). Cls.: number of target classes (range: 2 to 10; median: 2). Source: benchmark suite from which the task was drawn. CC18: OpenML-CC18; TA: TabArena; TL: Talent; TZ: TabZilla. Datasets shared across suites carry combined source tags (e.g., CC18,TZ).
ID 3 11 12 14 15 16 18 21 22 23 27 28 29 30 31 32 36 37 38 44 46 50 54 60 151 179 180 181 182 188 293 333 458 469 470 554 846 934 999 1038 1043 1046 1049 1050 1053 1063
Name kr-vs-kp balance-scale mfeat-factors mfeat-fourier breast-w mfeat-karhunen mfeat-morphological car mfeat-zernike cmc colic optdigits credit-approval page-blocks credit-g pendigits segment diabetes sick spambase splice tic-tac-toe vehicle waveform-5000 electricity adult covertype yeast satimage eucalyptus covertype monks-problems-1 analcatdata_authorship analcatdata_dmft profb mnist_784 elevators socmob audiology gina_agnostic ada_agnostic mozilla4 pc4 pc3 jm1 kc2
Samples
Feat.
Cls.
Source
3,196 625 2,000 2,000 699 2,000 2,000 1,728 2,000 1,473 368 5,620 690 5,473 1,000 10,992 2,310 768 3,772 4,601 3,190 958 846 5,000 45,312 48,842 110,393 1,484 6,430 736 581,012 556 841 797 672 70,000 16,599 1,156 226 3,468 4,562 15,545 1,458 1,563 10,885 522
37 5 217 77 10 65 7 7 48 10 23 65 16 11 21 17 20 9 30 58 61 10 19 41 9 15 55 9 37 20 55 7 71 5 10 785 19 6 70 971 49 6 38 38 22 22
2 3 10 10 2 10 10 4 10 3 2 10 2 5 2 10 7 2 2 2 3 2 4 3 2 2 7 10 6 5 2 2 4 6 2 10 2 2 2 2 2 2 2 2 2 2
CC18 CC18,TZ CC18 CC18,TZ CC18 CC18 CC18 TL CC18,TZ CC18 TZ CC18 CC18,TZ TL CC18,TZ CC18 TL CC18 CC18 CC18 CC18,TZ CC18 CC18,TZ TL CC18 TL TL TL CC18 CC18 TL TZ CC18 CC18 TZ CC18 TZ TZ TZ TL TZ TL CC18 CC18 CC18 CC18
Continued on next page 14
Ensembling Tabular Foundation Models
Table 4 (continued from previous page) ID 1067 1068 1111 1112 1114 1116 1119 1120 1169 1459 1461 1462 1464 1467 1468 1471 1475 1476 1478 1480 1485 1486 1487 1489 1494 1497 1501 1510 1565 1590 1596 4134 4534 4538 6332 23381 23512 23517 40536 40646 40647 40648 40649 40650 40668 40670 40680 40681 40682 40685 40701 40900 40945 40966 40975
Name kc1 pc1 KDDCup09_appetency KDDCup09_churn KDDCup09_upselling musk adult-census MagicTelescope airlines artificial-characters bank-marketing banknote-authentication blood-transfusion-service-center climate-model-simulation-crashes cnae-9 eeg-eye-state first-order-theorem-proving gas-drift har ilpd madelon nomao ozone-level-8hr phoneme qsar-biodeg wall-robot-navigation semeion wdbc heart-h adult covertype Bioresponse PhishingWebsites GesturePhaseSegmentationProcessed cylinder-bands dresses-sales higgs numerai28.6 SpeedDating GAMETES_Epistasis_2-Way_20atts_0.1H_EDM-1_1 GAMETES_Epistasis_2-Way_20atts_0.4H_EDM-1_1 GAMETES_Epistasis_3-Way_20atts_0.2H_EDM-1_1 GAMETES_Heterogeneity_20atts_1600_Het_0.4_0.2_50_EDM-2_001 GAMETES_Heterogeneity_20atts_1600_Het_0.4_0.2_75_EDM-2_001 connect-4 dna mofn-3-7-10 mux6 thyroid-new shuttle churn Satellite Titanic MiceProtein car
Samples
Feat.
Cls.
Source
2,109 1,109 50,000 50,000 50,000 6,598 32,561 19,020 539,383 10,218 45,211 1,372 748 540 1,080 14,980 6,118 13,910 10,299 583 2,600 34,465 2,534 5,404 1,055 5,456 1,593 569 294 48,842 581,012 3,751 11,055 9,873 540 500 98,050 96,320 8,378 1,600 1,600 1,600 1,600 1,600 67,557 3,186 1,324 128 215 58,000 5,000 5,100 1,309 1,080 1,728
22 22 231 231 231 168 16 12 8 8 17 5 5 21 857 15 52 129 562 11 501 119 73 6 42 25 257 31 14 15 55 1,777 31 33 40 13 29 22 121 21 21 21 21 21 43 181 11 7 6 10 21 37 14 82 7
2 2 2 2 2 2 2 2 2 10 2 2 2 2 9 2 6 6 6 2 2 2 2 2 2 4 10 2 5 2 7 2 2 5 2 2 2 2 2 2 2 2 2 2 3 3 2 2 3 7 2 2 2 8 4
CC18,TZ CC18 TL TL TL TL TL TL TZ TZ CC18 CC18 CC18 TL CC18,TZ TL CC18 TL CC18 CC18 CC18 CC18,TZ CC18 CC18 CC18,TZ CC18 CC18 CC18 TZ CC18 TL CC18,TZ CC18 CC18,TZ CC18 CC18 TZ CC18 TL TL TL TL TL TL CC18 CC18 TL TL TL TL CC18 TL TL CC18 CC18
Continued on next page
15
Ensembling Tabular Foundation Models
Table 4 (continued from previous page) ID 40978 40979 40981 40982 40983 40984 40994 41027 41138 41143 41147 41150 43945 43973 46905 46906 46908 46910 46911 46912 46913 46915 46916 46918 46919 46920 46921 46922 46924 46927 46929 46930 46932 46933 46935 46937 46938 46939 46940 46941 46947 46950 46952 46955 46956 46958 46960 46962 46963 46969 46979 46980
Name Internet-Advertisements mfeat-pixel Australian steel-plates-fault wilt segment climate-model-simulation-crashes jungle_chess_2pcs_raw_endgame_complete APSFailure jasmine albert MiniBooNE electricity phoneme Amazon_employee_access anneal APSFailure bank-marketing Bank_Customer_Churn Bioresponse blood-transfusion-service-center churn coil2000_insurance_policies credit-g credit_card_clients_default customer_satisfaction_in_airline diabetes Diabetes130US E-CommereShippingData Fitness_Club GiveMeSomeCredit hazelnut-spread-contaminant-detection heloc hiva_agnostic HR_Analytics_Job_Change_of_Data_Scientists in_vehicle_coupon_recommendation Is-this-a-good-customer kddcup09_appetency Marketing_Campaign maternal_health_risk online_shoppers_intention polish_companies_bankruptcy qsar-biodeg SDSS17 seismic-bumps splice students_dropout_and_academic_success taiwanese_bankruptcy_prediction website_phishing NATICUSdroid jm1 MIC
16
Samples
Feat.
3,279 2,000 690 1,941 4,839 2,310 540 44,819 76,000 2,984 425,240 130,064 38,474 3,172 32,769 898 76,000 45,211 10,000 3,751 748 5,000 9,822 1,000 30,000 129,880 768 71,518 10,999 1,500 150,000 2,400 10,459 3,845 19,158 12,684 1,723 50,000 2,240 1,014 12,330 5,910 1,054 78,053 2,584 3,190 4,424 6,819 1,353 7,491 10,885 1,699
1,559 241 15 28 6 20 21 7 171 145 79 51 9 6 10 39 171 14 11 1,777 5 20 86 21 24 22 9 48 11 7 11 31 24 1,618 13 25 14 213 26 7 18 65 42 12 16 61 37 95 10 87 22 112
Cls.
Source
2 CC18 10 CC18 2 TZ 7 CC18 2 CC18 7 CC18 2 CC18 3 CC18,TZ 2 TL 2 TZ 2 TZ 2 TZ 2 TZ 2 TZ 2 TA 5 TA 2 TA 2 TA 2 TA 2 TA 2 TA 2 TA 2 TA 2 TA 2 TA 2 TA 2 TA 2 TA 2 TA 2 TA 2 TA 2 TA 2 TA 3 TA 2 TA 2 TA 2 TA 2 TA 2 TA 3 TA 2 TA 2 TA 2 TA 3 TA 2 TA 3 TA 3 TA 2 TA 3 TA 2 TA 2 TA 8 TA