On the Learning Dynamics of Label-Noise Memorization in ReLU MLPs

Yannis Kaltampanidis, Mykola Pechenizkiy, Hannah Pinson

Keywords: Memorization, ReLU, MLP, Learning Dynamics

Abstract:

Understanding the mechanisms of memorization
in neural networks remains an open and challeng-
ing problem. In this work, we study label-noise
memorization in two-layer ReLU MLPs through
the learning dynamics of the first-layer weights1.

Our analysis suggests that label noise initially
attenuates first-layer magnitude evolution while
largely preserving weight directions, before in-
ducing competing magnitude and directional dy-
namics between clean and noisy samples. Experi-
ments on MNIST further suggest that these com-
peting dynamics can reach a dynamical equilib-
rium prior to memorization, indicating that memo-
rization can emerge without significant distortion
of the first-layer weights.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 62