Relational Representation Learning

Lucas Maes, Ian K Hajra, Arnesh Batra, Hugues Van Assel, Damien Scieur, Randall Balestriero

Track: Extended Abstract Track

Keywords: representation learning, foundation models, graph estimation

Abstract: We introduce Relational Representation Learning (RRL), a unifying paradigm casting representation learning as a graph estimation problem. Instead of treating samples in isolation, RRL defines learning objectives through a relational graph encoding pairwise relationships between data points. An encoder learns by estimating this graph from embeddings and minimizing its discrepancy with a specified target graph. This perspective reveals that self-, semi-, and supervised learning can all be recovered as special cases of RRL, providing a single formalism that consolidates diverse pretraining objectives into a unified mathematical object. This view offers a principled lens for analyzing empirical observations in self-supervised learning, such as slow convergence, performance, and the projector–backbone accuracy gap. Our experiments show that increasing relational richness within the graph improves convergence speed and downstream performance, while clarifying the role of auxiliary projection heads.

Submission Number: 111