Unveiling the Hessian's Connection to the Decision Boundary

Mahalakshmi Sabanayagam, Freya Behrens, Urte Adomaityte, Anna Dawid

Keywords: Hessian, decision boundary

TL;DR: Top eigenvectors of training loss Hessian encode the decision boundary learned by the neural network.

Abstract: Understanding the properties of well-generalizing minima is at the heart of deep learning research. On the one hand, the generalization of neural networks has been connected to the decision boundary complexity, which is hard to study in the high-dimensional input space. Conversely, the flatness of a minimum has become a controversial proxy for generalization. In this work, we provide the missing link between the two approaches and show that the Hessian top eigenvectors characterize the decision boundary learned by the neural network. Notably, the number of outliers in the Hessian spectrum is proportional to the complexity of the decision boundary. Based on this finding, we provide a new and straightforward approach to studying the complexity of a high-dimensional decision boundary.

Submission Number: 58