Spurious correlations in high dimensional regression: The roles of regularization, simplicity bias and over-parameterization

Learning models have been shown to rely on spurious correlations between non-predictive features and the associated labels in the training data, with negative implications on robustness, bias and fairness. In this work, we provide a statistical characterization of this phenomenon for high-dimensiona