A Light Bilevel Refinement Aligns Self-Supervised Representations for Stronger Task-Specific Learning

Self-supervised pretraining learns representations that are broadly transferable across downstream tasks, yet direct fine-tuning can be suboptimal due to misalignment between self-supervised and downstream task objectives, potentially degrading pretrained features beneficial to the downstream task.