Training shapes the curvature of shallow neural network representationsDownload PDF

Published: 07 Nov 2022, Last Modified: 05 May 2023NeurReps 2022 PosterReaders: Everyone
Keywords: Riemannian geometry, neural networks, curvature
TL;DR: We study how training shapes the Riemannian geometry induced by neural network feature maps.
Abstract: We study how training shapes the Riemannian geometry induced by neural network feature maps. At infinite width, shallow neural networks induce highly symmetric metrics on input space. Feature learning in networks trained to perform simple classification tasks magnifies local areas and reduces curvature along decision boundaries. These changes are consistent with previously proposed geometric approaches for hand-tuning of kernel methods to improve generalization.
7 Replies

Loading