An Analysis of Self-supervised Pre-training with Dependent Samples

Self-supervised learning relies on so-called data augmentations $ϕ(x)$ of unlabeled datapoints $x$ --- for example, masking random pixels in an image $x$ --- that should leave the label of $x$ invariant and are often used to learn a lower-complexity invariant subspace $\cal V$ for downstream tasks.