Can Contrastive Learning Refine Embeddings

Lihui Liu, Jinha Kim, Vidit Bansal

Abstract: Recent advancements in contrastive learning have revolutionized self-supervised representation learning and achieved state-of-the-art performance on benchmark tasks. While most existing methods focus on applying contrastive learning on input data modalities like images, natural language sentences, or networks, they overlook the potential of utilizing output from previously trained encoders. In this paper, we introduce SimSkip, a novel contrastive learning framework that specifically refines the input embeddings for downstream tasks. Unlike traditional unsupervised learning approaches, SimSkip takes advantage of the output embedding of encoder models as its input. Through theoretical analysis, we provide evidence that applying SimSkip does not lead to larger upper bounds on downstream task errors than that of the original embedding which is SimSkip’s input. Experiment results on various open datasets demonstrate that the embedding by SimSkip improves the performance on downstream tasks.