Huge Frozen Language Models as Readers for Open-Domain Question Answering

Yoav Levine, Ori Ram, Daniel Jannai, Barak Lenz, Shai Shalev-Shwartz, Amnon Shashua, Kevin Leyton-Brown, Yoav Shoham

Keywords: Language models, Open domain question answering, Retrieval.

TL;DR: We demonstrate that huge frozen LMs can reach and surpass leading fine tuning approaches for open-domain question answering.

Abstract: In the open-book variant of the open domain question answering setting, an answer generator typically attends to 100+ retrieved documents when answering, and is thus often called a "reader". Current readers are fine tuned for this long-context functionality. Because it is prohibitively expensive to fine tune huge models to attend to 100+ retrieved documents, readers tend to be relatively small, typically having fewer than 1B parameters. We introduce huge LMs into this pipeline as frozen readers. To do so, we use a re-ranking stage to condense relevant information from 100+ retrieved documents into the input sequence length of the frozen LM reader. We show that frozen LMs can reach and surpass leading fine tuning approaches on Natural Questions, a prominent open-domain question answering benchmark.