Beyond Structural Symmetries: Linear Mode Connectivity via Neuron Identifiability

Vincent Bürgin, Daniel Herbst, Ya-Wei Eileen Lin, Stefanie Jegelka

Keywords: Parameter symmetry, Linear mode connectivity, Identifiability, Symmetry breaking

TL;DR: We study data-dependent effective parameter symmetry breaking via neurons' induced function classes.

Abstract: Many striking phenomena in deep learning, such as linear mode connectivity and the structured behavior of training dynamics, are closely tied to parameter symmetries: transformations that leave the realized function unchanged. Despite growing attention to structural parameter symmetries, the exact interplay between parameters, data, and representations remains underexplored. To investigate this, we develop a theoretical framework of effective function classes defined by the neurons' induced functions restricted to the representation subspace. We then formalize effective symmetry breaking via neuron identifiability across independent training runs. Our analysis shows that neural networks can admit large families of approximately equivalent solutions even in structurally asymmetric models. We further show that neuron identifiability enables representation merging without prior alignment, and characterize when such merging admits a linear low-loss connecting path. These findings highlight the role of effective function classes in affecting the loss landscape.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 42