Early Period of Training Impacts Out-of-Distribution Generalization
Keywords: early period of training, out-of-distribution generalization, training dynamics
Abstract: Prior research has found that differences in the early period of neural network training significantly impact the performance of in-distribution (ID) tasks. However, trained neural networks are often sensitive to out-of-distribution (OOD) data, making them less reliable in downstream applications. Yet, the impact of the early training period on OOD generalization remains unknown due to its complexity and lack of effective analytical methodologies. In this work, we investigate the relationship between learning dynamics and OOD generalization during the early period of neural network training. We utilize the trace of Fisher Information and sharpness, focusing on gradual unfreezing (i.e., progressively unfreezing parameters during training) as the methodology for investigation. Through a series of empirical experiments, we show that 1) changing the number of trainable parameters at certain times (via gradual unfreezing) has minor impacts on ID results, but significantly improves OOD results; 2) the absolute values of sharpness and trace of Fisher Information at the initial period of training are not indicative for OOD generalization, a higher sharpness may be beneficial for OOD generalization in this period.
Student Paper: Yes
Submission Number: 10
Loading