HiCL: Hierarchical Contrastive Learning of Unsupervised Sentence Embeddings

Zhuofeng Wu, Chaowei Xiao, V.G.Vinod Vydiswaran

Submission Type: Regular Long Paper

Submission Track: Machine Learning for NLP

Keywords: Contrastive Learning, Semantic Textual Similarity, Hierarchical Training

TL;DR: A new training paradigm for textual contrastive learning that can seamless integrate with previous frameworks and improve them.

Abstract:

In this paper, we propose a hierarchical contrastive learning framework, HiCL, which considers local segment-level and global sequence-level relationships to improve training efficiency and effectiveness.
Traditional methods typically encode a sequence in its entirety for contrast with others, often neglecting local representation learning, leading to challenges in generalizing to shorter texts. Conversely, HiCL improves its effectiveness by dividing the sequence into several segments and employing both local and global contrastive learning to model segment-level and sequence-level relationships.
Further, considering the quadratic time complexity of transformers over input tokens, HiCL boosts training efficiency by first encoding short segments and then aggregating them to obtain the sequence representation.
Extensive experiments show that HiCL enhances the prior top-performing SNCSE model across seven extensively evaluated STS tasks, with an average increase of +0.2% observed on $BERT_{large}$ and +0.44% on $RoBERTa_{large}$.

Submission Number: 2688