Theoretical Grounds for Popularity Bias in Low-Rank Two-Tower Models Trained with Cosine-Based Loss

Keywords: Representation learning, Low-rank representations, Two-tower models, Embedding norms

TL;DR: Highly cited works (>960 citations) mischaracterize how embeddings move under cosine loss; we give new theory explaining that

Abstract: Low-rank two-tower models may exhibit popularity bias: a positive correlation between item frequency and embedding norm inflates dot-product scores for popular items, making magnitude dominate over directional similarity in the retrieval outcomes. This phenomenon is due to specific properties of loss and encoder architecture. We identify sufficient conditions under which embedding updates are provably orthogonal to the current embedding and hence monotonically increase its norm. Our theory yields explicit guarantees for the emergence of popularity bias in practical two-tower setups (InfoNCE loss, asymmetric two-tower architecture) and corrects a common misconception in prior works that orthogonality of the gradient alone implies norm inflation for deep encoders. Empirical studies support the theory via geometry-first investigation: configurations that satisfy these premises exhibit strictly orthogonal embedding movements and a robust statistically significant frequency–norm coupling, whereas violations of any premise break orthogonality and yield non-systematic update trajectories. The results provide theoretical grounds for popularity bias in cosine-trained two-tower models (particularly in recommender systems, though not limited to them) and show when it should be expected in production systems.

Submission Number: 20