Extreme Memorization via Scale of Initialization

Harsh Mehta, Ashok Cutkosky, Behnam Neyshabur

Keywords: Scale of initialization, Memorization, Overfitting, Generalization, Generalization Measure, Understanding Deep Learning

Abstract: We construct an experimental setup in which changing the scale of initialization strongly impacts the implicit regularization induced by SGD, interpolating from good generalization performance to completely memorizing the training set while making little progress on the test set. Moreover, we find that the extent and manner in which generalization ability is affected depends on the activation and loss function used, with sin activation being the most extreme. In the case of the homogeneous ReLU activation, we show that this behavior can be attributed to the loss function. Our empirical investigation reveals that increasing the scale of initialization correlates with misalignment of representations and gradients across examples in the same class. This insight allows us to device an alignment measure over gradients and representations which can capture this phenomenon. We demonstrate that our alignment measure correlates with generalization of deep models trained on image classification tasks.

One-sentence Summary: Through studying the effect of scale of initialization on generalization, we come up with an alignment measure that correlations with generalization of deep models.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Code: [ google-research/google-research](https://github.com/google-research/google-research/tree/master/extreme_memorization)

Data: [CIFAR-10](https://paperswithcode.com/dataset/cifar-10), [CIFAR-100](https://paperswithcode.com/dataset/cifar-100), [SVHN](https://paperswithcode.com/dataset/svhn)

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2008.13363/code)