On the Geometry of Memorization: Interpolation and Second-Order Representation Irregularity

Satwik Bathula

Keywords: Memorization, Representation Geometry, Interpolation, Second-Order Complexity, Curvature, Pullback Metric, Generalization

TL;DR: We show that memorization in neural networks is governed not by the magnitude of complexity but by the spatial organization of second-order geometric variation in learned representations.

Abstract: We study memorization through the second-order geometry of learned representations. Using the pullback metric, we show that interpolating rapidly varying labels forces large second-order variation, linking interpolation to higher-order complexity. Empirically, both structured and random labels exhibit large magnitude, but differ in their spatial organization. Structured targets yield smooth, globally distributed variation, while random targets produce sparse, high-magnitude spikes. This distinction is captured by a curvature localization measure, showing that memorization is not characterized by the magnitude of second-order variation, but by its organization in representation space.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 6