Low-Rank Networks Recover Weight and Functional Symmetry Better

Janis Aiad

Keywords: low-rank, neural-networks

TL;DR: Low rank neural networks acts as an implicit bias toward recovering functional and weight-space symmetry, while higher rank or full-rank models can fit the target without necessarily learning symmetric internal representations.

Abstract:

We study an empirical phenomenon in low-rank random-feature networks: even when stochastic training sees one non-symmetrized sample at a time, the learned model can recover a globally symmetric function and, in successful regimes, symmetric internal partial functions.
The effect is not explained by output invariance alone.
Low-loss counterexamples exist where the output is nearly symmetric but the learned partial functions remain asymmetric.
We therefore analyze symmetry simultaneously in function space, representation space, and weight space.
Across one-dimensional sum of cosines experiments and new multidimensional batch-size-one experiments, low-rank bottleneck networks exhibit a distinctive regime where symmetry is visible in late partial functions and in mirror-paired first-layer atoms with matched outgoing weights.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 30