DSReg: Using Distant Supervision as a Regularizer

Yuxian Meng, Muyu Li, Xiaoya Li, Wei Wu, Fei Wu, Jiwei Li

Abstract: In this paper, we aim at tackling a general issue in NLP tasks where some of the negative examples are highly similar to the positive examples, i.e., hard-negative examples). We propose the distant supervision as a regularizer (DSReg) approach to tackle this issue. We convert the original task to a multi-task learning problem, in which we first utilize the idea of distant supervision to retrieve hard-negative examples. The obtained hard-negative examples are then used as a regularizer, and we jointly optimize the original target objective of distinguishing positive examples from negative examples along with the auxiliary task objective of distinguishing soften positive examples (comprised of positive examples and hard-negative examples) from easy-negative examples. In the neural context, this can be done by feeding the final token representations to different output layers. Using this unbelievably simple strategy, we improve the performance of a range of different NLP tasks, including text classification, sequence labeling and reading comprehension.

Data: [NarrativeQA](https://paperswithcode.com/dataset/narrativeqa), [SST](https://paperswithcode.com/dataset/sst)