Emergence of Equivariance in Deep Ensembles

Jan E Gerken, Pan Kessel

Primary Area: learning theory

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: equivariant networks, deep ensembles, neural tangent kernel

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: Large-width deep ensembles are equivariant

Abstract: We demonstrate that a generic deep ensemble is emergently equivariant under data augmentation in the large width limit. Specifically, the ensemble is equivariant at any training step for any choice of architecture, provided that data augmentation is used. This equivariance also holds off-manifold and is emergent in the sense that predictions of individual ensemble members are not equivariant but their collective prediction is. As such, the deep ensemble is indistinguishable from a manifestly equivariant predictor. We prove this theoretically using neural tangent kernel theory and verify our theoretical insights using detailed numerical experiments.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 7253