Supervised Sentence Representation with Interactive Loss Decay

Anonymous

TL;DR: A novel interaction model is proposed on text representation tasks and achieves good results.

Abstract: Contrastive learning has achieved remarkable results in sentence representation, but its semantic representation remains independent in the process of training and inference, and could not pay attention to the interactive information of sentence pairs. This paper proposes InterCSE, a interactive contrastive learning method for sentence embedding, which not only focuses on the semantic similarity sorting of sentence pairs, but also increases the interactive information as a supplement for sentence representation. Meanwhile, we propose a loss decaying strategy to balance embedding similarity and interactive information objectives. We evaluate the performance of InterCSE on standard semantic textual similarity (STS) tasks, and experiments show that our model using $BERT_{base}$ and $BERT_{large}$ achieve 82.11\% and 82.88\% spearman's correlation, 0.54\% and 0.43\% improvement compared to SimCSE respectively. We also conduct experiments that adding interactive network based on Sentence-Transformers, which get 85.18\%(+0.88\% $BERT_{base}$) and 85.69\%(+1.07\% $RoBERTa_{base}$) spearman's correlation on STS Benchmark. Hence, adding interactive features to the traditional siamese network performs very well, and achieves new state-of-the-art performance on sentence representation tasks.

Paper Type: long

Research Area: Semantics: Sentence-level Semantics, Textual Inference and Other areas

Contribution Types: NLP engineering experiment, Reproduction study

Languages Studied: English