A Simple Angle-based Approach for Contrastive Learning of Unsupervised Sentence Representation
Abstract: Contrastive learning has been successfully adopted in the field of VRL (visual representation learning) by constructing contrastive pairs. After that, SRL (sentence representation learning) followed the literature established by the promising baseline SimCSE, which has made notable breakthroughs in unsupervised SRL. However, considering the difference between VRL and SRL, there is a still room for designing a novel contrastive framework specially targeted for SRL. We propose a novel angle-based similarity function for contrastive objective. By examining the gradient of our contrastive objective, we show that the angle-based similarity function provides better property for SRL in terms of training dynamics than the off-the-shelf cosine similarity: (1) effectively pulling a positive instance toward an anchor instance in the early stage of training and (2) not excessively repelling the false negative instance during the middle of training.
Our experimental results on widely-utilized benchmarks demonstrate the effectiveness and extensibility of our novel angle-based approach, and further analyses also back up the reason for its better sentence representation power.
Paper Type: Long
Research Area: Semantics: Lexical and Sentence-Level
Research Area Keywords: Sentence Representation Learning, Unsupervised Contrastive Learning, Semantic Textual Similarity
Contribution Types: NLP engineering experiment
Languages Studied: English
Submission Number: 322
Loading