SimCSE++: Improving Contrastive Learning for Sentence Embeddings from Two Perspectives

Jiahao Xu, Wei Shao, Lihui Chen, Lemao Liu

Submission Type: Regular Long Paper

Submission Track: Semantics: Lexical, Sentence level, Document Level, Textual Inference, etc.

Keywords: sentence embedding, contrastive learning, dimention-wise contrastive learning

TL;DR: This paper improve contrastive learning for sentence embeddings by handling dropout noise in negatives and dimension-wise contrastive learning regulation.

Abstract: This paper improves contrastive learning for sentence embeddings from two perspectives: handling dropout noise and addressing feature corruption. Specifically, for the first perspective, we identify that the dropout noise from negative pairs affects the model's performance. Therefore, we propose a simple yet effective method to deal with such type of noise. Secondly, we pinpoint the rank bottleneck of current solutions to feature corruption and propose a dimension-wise contrastive learning objective to address this issue. Both proposed methods are generic and can be applied to any contrastive learning based models for sentence embeddings. Experimental results on standard benchmarks demonstrate that combining both proposed methods leads to a gain of 1.8 points compared to the strong baseline SimCSE configured with BERT base. Furthermore, applying the proposed method to DiffCSE, another strong contrastive learning based baseline, results in a gain of 1.4 points.

Submission Number: 113