Is Memorization Actually Necessary for Generalization?

Hadi Abdullah, Ke Wang, Blaine Hoak, Yizhen Wang, Sunpreet S. Arora, Yiwei Cai

Primary Area: general machine learning (i.e., none of the above)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: memorization, subpopulations, influence functions

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: Memorization does not impact generalization, as perviously claimed.

Abstract: Memorization is the ability of deep models to associate training data with seemingly random labels. Even though memorization may not align with models’ ability to generalize, recent work by Feldman and Zhang (2020) has demonstrated that memorization is in fact necessary for generalization. However, upon closer inspection of this work, we uncover several methodological errors including lack of model convergence, data leakage, and sub- population shift. We show that these errors led to a significant overestimation of memorization’s impact on test accuracy (by over five times). After accounting for these errors, we demonstrate that memorization does not impact prediction accuracy as previously reported, and therefore, it is not necessary for generalization. In light of these findings, future researchers are encouraged to design better techniques to identify memorized points that can avoid some of the earlier stated problems.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 5660