When Does Self-Supervised Pretraining Help Tabular Models? A Study of Label Scarcity and Missing Data

In this work, we conduct an empirical study on self-supervised learning (SSL) for tabular data. Our experiments examine limited label regimes (that is, 1%, 5%, 10%, and 20% of the labels) and highlight limitations and opportunities in current tabular SSL. Contrary to what is expected, our mask-and-r