SCONER: Scoring Negative Candidates\\Before Training Neural Re-Ranker For Question Answering

Man Luo, Mihir Parmar, Jayasurya Sevalur Mahendran, Sahit Jain, Samarth Rawal, Chitta Baral

Keywords: Neural re-ranker, question answering

Abstract:

A neural re-ranker aims to re-scores a set of candidates given by a search engine.
It is crucial to obtain good performance on many down-stream tasks such as retrieval-based question answering (ReQA).
In this work, we introduce a scoring function for negative candidates to train a neural re-ranker and compare models trained by our approach with three baselines on a range of ReQA tasks.
We term our approach as SCONER---scoring negative candidates before training neural re-ranker, which includes 1) a scoring function based on the concept of Semantic Textual Similarity (STS) and data augmentation; and 2) a neural re-ranker trained on data using generated negativeness scores as labels.
Experimental results show that SCONER outperforms three baselines by up to 13\% absolute improvement on the SearchQA dataset and 5.5\% on average across all datasets in terms of P@1. SCONER demonstrates that using different negativeness scores to train a neural-ranker is better than a single score, and we present a simple yet efficient way to generate the scores.