Memorization is Not Learning: Delineated through Features and Labels

Rishi Singhal, Jung-Eun Kim

Keywords: Feature Memorization, Feature learning, Label memorization

Abstract: Although deep learning is widely adopted for its capability to fit training data effectively, it often memorizes outliers and/or mislabeled instances, a phenomenon known as label memorization. As for features, while prior studies have examined how features influence model behavior, the distinction between feature memorization and feature learning still remains opaque or ambiguous. Moreover, the intricate relationship between feature memorization and learning with label memorization is not well understood. Hence, in this work, our contribution is twofold: First, we precisely distinguish memorization and learning at the feature level and define conditions under which they occur. Second, we investigate the interactions among feature memorization, feature learning, and label memorization, revealing that label memorization suppresses feature memorization while causing feature learning. These findings offer new insights into memorization and learning in neural networks.

Primary Area: interpretability and explainable AI

Submission Number: 11676