A Simple Finetuning Strategy Based on Bias-Variance Ratios of Layer-Wise Gradients

Finetuning is an effective method for adapting pretrained networks to downstream tasks. However, the success of finetuning de pends heavily on the selection of layers to be tuned, as full finetuning can lead to overfitting, while tuning only the last layer may not capture the necessary task-specific features. This requires a balanced approach of automatic layer selection to achieve higher performance. In this con text, we propose the Bias-Variance Guided Layer Selection (BVG-LS), a simple yet effective strategy that adaptively selects a layer to be tuned at each training iteration. More specifically, BVG-LS computes the bias variance ratios of mini-batch gradients for each layer and updates the parameters of the layer with the largest ratio. This strategy reduces the risk of overfitting while maintaining the model’s capacity to learn task specific features. In our experiments, we demonstrate the effectiveness of the BVG-LS strategy on seven image classification tasks. We show that BVG-LS outperforms full finetuning on all tasks with the WideResNet 50-2 model and on six out of seven tasks with the ViT-S model.