Parameter symmetries determine representational geometry in overparameterized nonlinear networks

Marvin Theiss, Lukas Braun, Andrew M Saxe, Erin Grant

Keywords: parameter symmetries, representational geometry, identifiability, overparameterization, deep learning theory

Abstract: Representations are routinely used in machine learning, psychology, and neuroscience to probe the computations of biological and artificial systems. Yet it remains unclear to what extent computation constrains representation in artificial neural networks. Neural networks admit *parameter symmetries*: transformations of the parameters that preserve function exactly but reshape representational geometry. We show that known parameter symmetries act on representations through just three primitives: *addition*, *duplication*, and *scaling*, giving a closed-form descriptor of representational geometry as task-linked features plus symmetry-induced noise. This decomposition yields analytic bounds on representational similarity between networks related by parameter symmetry, which reveal when functionally equivalent networks become arbitrarily *dis*similar. Finally, we identify *privileged* representational geometries, which weight features by their computational importance and recover a stable link between representation and computation. Overall, our results delineate when representation can support inferences about computation, and when it cannot.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 48