On Rotational Symmetry in the Loss landscape of Self-Supervised Learning

Liu Ziyin, Ekdeep Singh Lubana, Masahito Ueda, Hidenori Tanaka

Keywords: self-supervised learning, collapse, symmetry breaking, phase transition

TL;DR: We analytically solve the loss landscape of self-supervised learning and identify the causes of complete and dimensional collapse to be relevant to symmetry breaking

Abstract: We derive an analytically tractable theory of SSL landscape and show that it accurately captures an array of collapse phenomena and identifies their causes.