Implicitly Learned Invariance and Equivariance in Linear Regression
Keywords: Invariance, equivariance, symmetry learning, implicit learning
TL;DR: We find that linear regression models can learn symmetries in their data, without them being given as priors.
Abstract: Can deep learning models generalize if their problem's underlying structure is unknown a priori? We analyze this theoretically and empirically in an idealistic setting for linear regression with invariant/equivariant data. We prove that linear regression models learn to become invariant/equivariant, with their weights being decomposed into a component that respects the symmetry and one that does not. These two components evolve independently over time, with the asymmetric component decaying exponentially given sufficient data. Extending these results to complex systems will be pursued in future work.
Type Of Submission: Extended Abstract (4 pages, non-archival)
Submission Number: 66
Loading