Geodesic Mode ConnectivityDownload PDF

01 Mar 2023 (modified: 30 May 2023)Submitted to Tiny Papers @ ICLR 2023Readers: Everyone
Keywords: geometry, information geometry, mode connectivity, geodesic, generalization
TL;DR: We explore mode connectivity, the phenomenon of trained models being connected by a path of low loss, in the context of geodesics in distribution space.
Abstract: Mode connectivity is a phenomenon where trained models are connected by a path of low loss. We reframe this in the context of Information Geometry, where neural networks are studied as spaces of parameterized distributions with curved geometry. We hypothesize that shortest paths in these spaces, known as geodesics, correspond to mode-connecting paths in the loss landscape. We propose an algorithm to approximate geodesics and demonstrate that they achieve mode connectivity.
6 Replies