Adversarially Trained Linear Transformers Are Optimal Robust In-Context Learners for Gaussian Mixtures

Adversarial training is one of the most reliable defenses against adversarial attacks, but its high computational cost must generally be paid anew for each task. Robust foundation models offer a promising alternative: adversarially pretrain a model once and then transfer its robustness to downstream