A second-order perspective on linear mode connectivity modulo permutations

Beatrix Benkő

Track: Extended Abstract Track

Keywords: Linear mode connecivity, permutation alignment, second-order approximations

TL;DR: We assess the exactness of loss and network output approximations during interpolation in parameter space.

Abstract:

Linear mode connectivity of neural networks revolves around the striking observation that solutions obtained from the same initialization can often be connected by linear paths in parameter space, without substantial loss increase along them. Recent advances have demonstrated that accounting for permutations of hidden units enables linear mode connectivity even between independently trained networks. We provide insights into this phenomenon from a second-order perspective, employing local, derivative-based approximations of the loss and network outputs.
We discover that such approximations systematically underestimate the rise in loss between independently trained minima, yet they can accurately capture the loss behavior along the path between the permutation-aligned solutions. We attribute the failure to higher-order variations in loss arising when interpolating misaligned units, which are eliminated by permutation reordering, while the remaining non-linear effects diminish with increasing network width. Beneath the aggregate loss values, however, we find that the network outputs and the prediction transitions they induce often evolve in a less easily traceable manner.

Submission Number: 147