Back to Fundamentals: Re-Examining Memorization in Deep Learning Models
Keywords: Memorization
TL;DR: Understanding memorization in ML models
Abstract: In supervised training, memorization is the ability of deep learning models to assign arbitrary ground truth labels to inputs in the dataset. Due to the computa- tional difficulty of identifying existing memorized points, researchers often induce artificial memorization i.e, force the model to memorize the newly introduced points (via Noisy Label or Noisy Input). However, in this work, we show that this artificial proxy exhibits fundamentally different characteristics than the mem- orization real points (or natural memorization). To demonstrate this deviation, we re-examine two key findings derived from artificial memorization and com- pare them against natural memorization i.e., over-parametrization and increased training time increases memorization. We show that both these factors have the opposite effect i.e., they reduce natural memorization. Additionally, we find that memorization and train-test gap are strongly correlated (Pearson score 0.99). As a result, memorization is not necessary for generalization. Since real world models suffer from natural memorization (instead of the artificial one) our findings sug- gest the research community should focus on natural memorization, instead of the artificial proxy.
Primary Area: alignment, fairness, safety, privacy, and societal considerations
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 3182
Loading