Boot and Switch: Alternating Distillation for Zero-Shot Dense Retrieval

Fan Jiang, Qiongkai Xu, Tom Drummond, Trevor Cohn

Submission Type: Regular Long Paper

Submission Track: Information Retrieval and Text Mining

Keywords: dense retrieval, iterated learning, alternating distillation, bootstrapping

TL;DR: A novel alternating bootstrapping training framework for zero-shot dense retrieval.

Abstract:

Neural 'dense' retrieval models are state of the art for many datasets, however these models often exhibit limited domain transfer ability.
Existing approaches to adaptation are unwieldy, such as requiring explicit supervision, complex model architectures, or massive external models.
We present $\texttt{ABEL}$, a simple but effective unsupervised method to enhance passage retrieval in zero-shot settings.
Our technique follows a straightforward loop: a dense retriever learns from supervision signals provided by a reranker, and subsequently, the reranker is updated based on feedback from the improved retriever.
By iterating this loop, the two components mutually enhance one another's performance.
Experimental results demonstrate that our unsupervised $\texttt{ABEL}$ model outperforms both leading supervised and unsupervised retrievers on the BEIR benchmark.
Meanwhile, it exhibits strong adaptation abilities to tasks and domains that were unseen during training.
By either fine-tuning $\texttt{ABEL}$ on labelled data or integrating it with existing supervised dense retrievers, we achieve state-of-the-art results.\footnote{Source code is available at \url{https://github.com/Fantabulous-J/BootSwitch}.}

Submission Number: 1266