Simple Temperature Cool-down in Contrastive Framework for Unsupervised Sentence Representation Learning

Anonymous

Abstract: In this paper, we proposes a simple, tricky method to improve sentence representation of unsupervised contrastive learning. Even though contrastive learning has achieved great performances in both visual representation learning (VRL) and sentence representation learning (SRL) fields, we focus on the fact that there is a gap between characteristics and training dynamics of VRL and SRL. We first examine the role of temperature to bridge the gap between VRL and SRL, and find some temperature-dependent elements in SRL; \textit{i.e.}, a higher temperature causes overfitting of the uniformity while improving the alignment in earlier phase of training. Then, we design a \textit{temperature cool-down} technique based on this observation, which helps PLMs to be more suitable for contrastive learning via preparation of uniform representation space. Our experimental results on widely-utilized benchmarks demonstrate the effectiveness and extensiblity of our method.

Paper Type: short

Research Area: Semantics: Sentence-level Semantics, Textual Inference and Other areas

Contribution Types: NLP engineering experiment

Languages Studied: English

Consent To Share Submission Details: On behalf of all authors, we agree to the terms above to share our submission details.