Do We Need to Differentiate Negative Candidates Before Training a Neural Ranker?

Anonymous

Abstract: Retrieval-based Question Answering (ReQA) requires a system to find candidates (e.g., sentences or short passages) containing the answer to a given question from a large corpus. A promising way to solve this task is a two-stage pipeline, where the first stage retrieves a set of candidates, and the second stage uses a neural network to rank the retrieved candidates. There are three standard methods to train neural rankers, Binary Cross-Entropy loss, Mean Square Error loss, and Hinge loss. While all these training strategies assign the same label for all the negative candidates, we argue that negativeness is not binary but exists as a spectrum, i.e., some candidates may be more negative than the others, and thus should be treated differently. We present SCONER---scoring negative candidates before training neural ranker---a model trained to differentiate negative candidates. Our approach includes 1) semantic textual similarity-based scoring together with data augmentation for score generation of negative candidates, and 2) a neural ranker trained on data using generated scores as labels. Together, we systematically compare three standard training methods and our proposed method on a range of ReQA datasets under multiple settings (i.e., single-domain and multi-domain). Our finding suggests that using more negative candidates to train neural rankers are better than less in both single- and multi-domain settings, where SCONER is the best in the single-domain settings and Hinge loss is the best in multi-domain settings.