Learning to Perturb for Contrastive Learning of Unsupervised Sentence Representations

Kun Zhou, Yuanhang Zhou, Xin Zhao, Ji-Rong Wen

Keywords: Unsupervised Sentence Representations, Contrastive Learning

Abstract:

Recently, contrastive learning has been shown effective in fine-tuning pre-trained language models (PLM) to learn sentence representations, which incorporates perturbations into unlabeled sentences to augment semantically related positive examples for training.
However, previous works mostly adopt heuristic perturbation methods that are independent of the sentence representations.
Since the perturbations are unaware of the goal or process of sentence representation learning during training, it is likely to lead to sub-optimal augmentations for conducting constrative learning.
To address this issue, we propose a new framework \textbf{L2P-CSR} that adopts a learnable perturbation strategy for improving contrastive learning of sentence representations.
In our L2P-CSR, we design a safer perturbation mechanism that only weakens the influence of tokens and features on the sentence representation, which avoids dramatically changing the semantics of the sentence representations.
Besides, we devise a gradient-based algorithm to generate adaptive perturbations specially for the dynamically updated sentence representation during training.
Such a way is more capable of augmenting high-quality examples that guide the sentence representation learning.
Extensive experiments on diverse sentence-related tasks show that our approach outperforms competitive baselines.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Applications (eg, speech processing, computer vision, NLP)