Finding Symmetry in Neural Network Parameter Spaces

Bo Zhao, Nima Dehmamy, Robin Walters, Rose Yu

Keywords: Parameter space symmetry

Abstract: Parameter space symmetries, or loss-invariant transformations, are important for understanding neural networks' loss landscape, training dynamics, and generalization. However, identifying the full set of these symmetries remains a challenge. In this paper, we formalize data-dependent parameter symmetries and derive their infinitesimal form, which enables an automated approach to discover symmetry across different architectures. Our framework systematically uncovers parameter symmetries, including previously unknown ones. We also prove that symmetries in smaller subnetworks can extend to larger networks, enabling direct generalization of discovered symmetries to more complex models.

Primary Area: learning theory

Submission Number: 23538