Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task LearningDownload PDF

15 Feb 2018 (modified: 21 Apr 2024)ICLR 2018 Conference Blind SubmissionReaders: Everyone
Abstract: A lot of the recent success in natural language processing (NLP) has been driven by distributed vector representations of words trained on large amounts of text in an unsupervised manner. These representations are typically used as general purpose features for words across a range of NLP problems. However, extending this success to learning representations of sequences of words, such as sentences, remains an open problem. Recent work has explored unsupervised as well as supervised learning techniques with different training objectives to learn general purpose fixed-length sentence representations. In this work, we present a simple, effective multi-task learning framework for sentence representations that combines the inductive biases of diverse training objectives in a single model. We train this model on several data sources with multiple training objectives on over 100 million sentences. Extensive experiments demonstrate that sharing a single recurrent sentence encoder across weakly related tasks leads to consistent improvements over previous methods. We present substantial improvements in the context of transfer learning and low-resource settings using our learned general-purpose representations.
TL;DR: A large-scale multi-task learning framework with diverse training objectives to learn fixed-length sentence representations
Keywords: distributed sentence representations, multi-task learning
Code: [![github](/images/github_icon.svg) facebookresearch/SentEval](https://github.com/facebookresearch/SentEval) + [![Papers with Code](/images/pwc_icon.svg) 3 community implementations](https://paperswithcode.com/paper/?openreview=B18WgG-CZ)
Data: [BookCorpus](https://paperswithcode.com/dataset/bookcorpus), [GLUE](https://paperswithcode.com/dataset/glue), [MRPC](https://paperswithcode.com/dataset/mrpc), [MultiNLI](https://paperswithcode.com/dataset/multinli), [Quora Question Pairs](https://paperswithcode.com/dataset/quora-question-pairs), [SNLI](https://paperswithcode.com/dataset/snli), [SentEval](https://paperswithcode.com/dataset/senteval)
Community Implementations: [![CatalyzeX](/images/catalyzex_icon.svg) 7 code implementations](https://www.catalyzex.com/paper/arxiv:1804.00079/code)
20 Replies