Random Label Prediction Heads for Studying and Controlling Memorization in Deep Neural Networks

ICLR 2026 Conference Submission12749 Authors

Keywords: Memorization, Random Labels, Overfitting, Generalization, Regularization

Abstract:

We introduce a straightforward yet effective method to empirically measure and regularize memorization in deep neural networks for classification tasks.
Our approach augments each training sample with random auxiliary labels, which are then predicted by a random label prediction head (RLP-head).
RLP-heads can be attached at arbitrary depths of a network, predicting random labels from the corresponding intermediate representation and thereby enabling analysis of how memorization capacity evolves across layers.
By interpreting RLP-head performance as an empirical estimate of Rademacher complexity, we obtain a direct measure of both sample-level memorization and model capacity.
We leverage this random label accuracy metric to analyze generalization and overfitting in different models and datasets.
Building on this approach, we further propose a novel regularization technique based on the output of the RLP-head, which demonstrably reduces memorization.
Interestingly, our experiments reveal that reducing memorization can either improve or impair generalization, depending on the dataset and training setup.
These findings challenge the traditional assumption that overfitting is equivalent to memorization and suggest new hypotheses to reconcile these seemingly contradictory results.

Primary Area: other topics in machine learning (i.e., none of the above)

Submission Number: 12749