A simple connection from loss flatness to compressed neural representations

Shirui Chen, Stefano Recanatesi, Eric Todd SheaBrown

Keywords: Sharpness, Flatness, compression, robustness

TL;DR: Compression is characterized by sharpness.

Abstract: Sharpness, a geometric measure in the parameter space that reflects the flatness of the loss landscape, has long been studied for its potential connections to neural network behavior. While sharpness is often associated with generalization, recent work highlights inconsistencies in this relationship, leaving its true significance unclear. In this paper, we investigate how sharpness influences the local geometric features of neural representations in feature space, offering a new perspective on its role. We introduce this problem and study the Maximum Local Sensitivity (MLS) and Network MLS (NMLS), measuring the network output w.r.t. internal neural representations. We show that MLS/NMLS correlate with the flatness of the loss around the local minima; and that this correlation is predicted by a relatively simple mathematical relationship: a flatter loss corresponds to a lower upper bound on the compression metrics of neural representations. Our work builds upon the linear stability trick by Ma and Ying, deriving inequalities between various compression metrics and quantities involving sharpness. Our inequalities readily extend to reparametrization-invariant sharpness as well. Through empirical experiments on various feedforward, convolutional, and transformer architectures, we find that our inequalities predict a consistently positive correlation between local representation compression and sharpness.

Student Paper: Yes

Submission Number: 59