Quantifying Symmetries: How Optimisers Impact the Functional Dimension

Johanna Marie Gegenfurtner, Naima Elosegui Borras, Georgios Arvanitidis

Keywords: hidden symmetries, ReLU networks, optimisation

TL;DR: We investigate which optimisers are implicitly biased towards parameter solutions with a larger number of hidden symmetries.

Abstract:

Overparameterised neural networks exhibit extensive symmetries in the parameter space. We
investigate which optimisers are implicitly biased
towards parameter solutions with a larger number
of hidden symmetries. We provide both theoretical results, and study this in regression experiments by evaluating the functional dimension
(the number of independent directions in the pa-
rameter space along which the network changes),
that serves as an empirical tool to evaluate the
prevalence of hidden symmetries. Moreover, we
relate our results on functional dimension to flatness metrics involving the Hessian of the loss with
respect to the parameters.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 26