On the memorisation of image classifiers

Michal Lukasik, Vaishnavh Nagarajan, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar

Keywords: memorization, generalization, distillation

Abstract: The success of modern neural models has prompted study of the connection between memorisation and generalisation: such models generalise well, despite being able to perfectly fit (“memorise”) completely random labels. To more carefully study this issue, recent work proposed an intuitive metric to quantify the degree of memorisation of individual training examples, and empirically computed the corresponding memorisation profile of a ResNet on image classification benchmarks. While an exciting first glimpse into what real-world models memorise, these studies leave open a fundamental question: do larger neural models memorise more? We present a comprehensive empirical analysis of this question on image classification benchmarks. Intriguingly, we find that training examples exhibit a diverse set of memorisation trajectories across model sizes: while most samples experience decreased memorisation under larger models, surprisingly, there is a dichotomy between the remaining samples. In particular, per-example memorisation trajectories reveal that examples exhibit decreasing or cap-shaped memorisation, and some examples even exhibit increasing memorisation. We further show that various memorisation proxies fail to capture such fundamental characteristics as we vary model size. Lastly, we find that knowledge distillation — an effective and popular model compression technique — tends to inhibit memorisation, while also improving generalisation. Intriguingly, we find that the memorisation is mostly inhibited on examples for which memorisation increases as the model size increases, thus pointing at how distillation improves generalisation by limiting memorisation of such examples.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 8106