Identity Crisis: Memorization and Generalization Under Extreme Overparameterization

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Michael C. Mozer, Yoram Singer

Abstract:

We study the interplay between memorization and generalization of
overparametrized networks in the extreme case of a single training example.
The learning task is to predict an output which is as similar as possible to
the input. We examine both fully-connected and convolutional networks that are
initialized randomly and then trained to minimize the reconstruction
error. The trained networks take one of the two forms: the constant
function (``memorization'') and the identity function (``generalization''). We
show that different architectures exhibit vastly different inductive bias
towards memorization and generalization.