Variational Open-domain Question Answering

Valentin Liévin, Andreas Geert Motzfeldt, Ida Riis Jensen, Ole Winther

Abstract: Retrieval-augmented models have proven to be effective in natural language processing tasks, yet there remains a lack of research on their opti- mization using variational inference. We intro- duce the Variational Open-Domain (VOD) frame- work for end-to-end training and evaluation of retrieval-augmented models, focusing on open- domain question answering and language mod- elling. The VOD objective, a self-normalized es- timate of the Rényi variational bound, is a lower bound to the task marginal likelihood and evalu- ated under samples drawn from an auxiliary sam- pling distribution (cached retriever and/or approx- imate posterior). It remains tractable, even for re- triever distributions defined on large corpora. We demonstrate VOD’s versatility by training reader- retriever BERT-sized models on multiple-choice medical exam questions. On the MedMCQA dataset, we outperform the domain-tuned Med- PaLM by +5.3% despite using 2.500× fewer pa- rameters. Our retrieval-augmented BioLinkBERT model scored 62.9% on the MedMCQA and 55.0% on the MedQA-USMLE. Last, we show the effectiveness of our learned retriever compo- nent in the context of medical semantic search.