Supervised Contrastive Learning for Pre-trained Language Model Fine-tuningDownload PDF

28 Sep 2020 (modified: 25 Jan 2021)ICLR 2021 PosterReaders: Everyone
  • Keywords: pre-trained language model fine-tuning, supervised contrastive learning, natural language understanding, few-shot learning, robustness, generalization
  • Abstract: State-of-the-art natural language understanding classification models follow two-stages: pre-training a large language model on an auxiliary task, and then fine-tuning the model on a task-specific labeled dataset using cross-entropy loss. Cross-entropy loss has several shortcomings that can lead to sub-optimal generalization and instability. Driven by the intuition that good generalization requires capturing the similarity between examples in one class and contrasting them with examples in other classes, we propose a supervised contrastive learning (SCL) objective for the fine-tuning stage. Combined with cross-entropy, the SCL loss we propose obtains significant improvements over a strong RoBERTa-Large baseline on multiple datasets of the GLUE benchmark in the few-shot learning settings, and it does not require any specialized architecture, data augmentation of any kind, memory banks, or additional unsupervised data. The new objective leads to models that are more robust to different levels of noise in the training data, and can generalize better to related tasks with limited labeled data.
  • Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics
11 Replies