Can Models Learn From Arbitrary Pairs?

Jiantao Wu, Sara Atito, Shentong Mo, Josef Kittler, Zhenhua Feng, Muhammad Awais

Keywords: representation learning, contrastive learning, supervised learning

TL;DR: SimLAP can learn from arbitrary pairs of classes robustly and promote distinct pairs close in subspaces while preserving class separability in global space

Abstract:

Representation learning traditionally follows a simple principle: pull semantically similar samples together and push dissimilar ones apart.
This principle underlies most existing approaches, including supervised classification, self-supervised learning, and contrastive methods, and it has been central to their success. Yet it overlooks an important source of information: Even when classes appear unrelated, their samples often share latent visual attributes such as shapes, textures, or structural patterns. For example, cats, dogs and cattle have fur and four limbs etc. These overlooked commonalities raise a fundamental question: *can models learn from arbitrary pairs without explicit guidance?*

We show that the answer is yes. The primary challenge lies in learning from dissimilar samples while preserving the notion of semantic distance. We resolve this by proving that for any pair of classes, there exists a subspace where their shared features are discriminative to other classes.
To uncover these subspaces we propose **SimLAP**, a **Sim**ple framework to **L**earn from **A**rbitrary **P**air. SimLAP uses a lightweight feature filter to adaptively activate shared attributes for any given pair.
Through extensive experiments we show that models trained via SimLAP can indeed learn effectively from arbitrary pairs.
Remarkably, models learned from arbitrary pairs are more transferable than those learned from traditional representation learning methods and exhibit greater resistance to representation collapse.
Our findings suggest that arbitrary pairs, often dismissed as irrelevant, are in fact a rich, complementary and untapped source of supervision. By learning from them we move beyond rigid notions of similarity. Hopefully, SimLAP will open an additional pathway toward more general and robust representation learning.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 7405