What Should We Freeze? Guarded Freezing: Connectivity Shapes the Fine-Tuning of Pretrained Models

When adapting pre-trained models through fine-tuning, freezing weights alone might not preserve performance, as updates elsewhere can change the inputs to the frozen core, ultimately affecting overall performance. We first analyse the case where a selected frozen core can be isolated and propose rem