The Hidden Pitfalls of the Cosine Similarity Loss

Andrew Draganov, Sharvaree Vadgama, Erik J Bekkers

Keywords: Self-Supervised Learning, Cosine Similarity, Gradient Dynamics

TL;DR: The cosine similarity can only be optimized on embeddings with small norms but optimizing the cosine similarity grows the embedding norms.

Abstract: We show that the gradient of the cosine similarity between two points goes to zero in two unexpected settings: (1) if a point has large magnitude or (2) if the points are on opposite ends of the latent space. Counterintuitively, we prove that optimizing the cosine similarity between points forces them to grow in magnitude. Thus, (1) is unavoidable in practice. We then observe that these derivations are extremely general -- they hold across deep learning architectures and for many of the standard self-supervised learning (SSL) loss functions. This leads us to propose cut-initialization: a simple change to network initialization that helps all studied SSL methods converge faster.

Student Paper: Yes

Submission Number: 38