Lazy vs hasty: linearization in deep networks impacts learning schedule based on example difficulty

26 Sept 2022, 15:15 (modified: 27 Jan 2023, 19:42)Accepted by TMLREveryoneRevisionsBibTeX
Abstract: Among attempts at giving a theoretical account of the success of deep neural networks, a recent line of work has identified a so-called `lazy' training regime in which the network can be well approximated by its linearization around initialization. Here we investigate the comparative effect of the lazy (linear) and feature learning (non-linear) regimes on subgroups of examples based on their difficulty. Specifically, we show that easier examples are given more weight in feature learning mode, resulting in faster training compared to more difficult ones. In other words, the non-linear dynamics tends to sequentialize the learning of examples of increasing difficulty. We illustrate this phenomenon across different ways to quantify example difficulty, including c-score, label noise, and in the presence of easy-to-learn spurious correlations. Our results reveal a new understanding of how deep networks prioritize resources across example difficulty.
License: Creative Commons Attribution 4.0 International (CC BY 4.0)
Submission Length: Regular submission (no more than 12 pages of main content)
Changes Since Last Submission: In figure 2 top left we now plot only 4 bins (top-10%, bottom-10% and 2 bins inbetween) in order to make the figure easier to read. We also fixed the inconsistent spelling of "Figure".
Assigned Action Editor: ~Jaehoon_Lee2
Submission Number: 461