Optimizing Upstream Representations for Out-of-Domain Detection with Supervised Contrastive Learning

Bo Wang, Tsunenori Mine

Abstract: Out-of-Domain (OOD) text detection has attracted significant research interest. However, conventional approaches primarily employ Cross-Entropy loss during upstream encoder training and seldom focus on optimizing discriminative In-Domain (IND) and OOD representations. To fill this gap, we introduce a novel method that applies supervised contrastive learning (SCL) to IND data for upstream representation optimization. This effectively brings the embeddings of semantically similar texts together while pushing dissimilar ones further apart, leading to more compact and distinct IND representations. This optimization subsequently improves the differentiation between IND and OOD representations, thereby enhancing the detection effect in downstream tasks. To further strengthen the ability of SCL to consolidate IND embedding clusters, and to improve the generalizability of the encoder, we propose a method that generates two different variations of the same text as "views". This is achieved by applying a twice "dropped-out" on the embeddings before performing SCL. Extensive experiments indicate that our method not only outperforms state-of-the-art approaches, but also reduces the requirement for training a large 354M-parameter model down to a more efficient 110M-parameter model, highlighting its superiority in both effectiveness and computational economy.