Zero-Shot Dense Retrieval with Momentum Adversarial Domain Invariant RepresentationDownload PDF

29 Sept 2021 (modified: 13 Feb 2023)ICLR 2022 Conference Withdrawn SubmissionReaders: Everyone
Keywords: dense retrieval, zero-shot, unsupervised domain adaptation
Abstract: Dense retrieval (DR) methods conduct text retrieval by first encoding texts in the embedding space and then matching them by nearest neighbor search. This requires strong locality properties from the representation space, i.e, the close allocations of each small group of relevant texts, which is hard to generalize to domains without sufficient training data. In this paper, we aim to improve the generalization ability of DR models from source training domains with rich supervision signals to target domains without any relevant labels, in the zero-shot setting. To achieve that, we propose Momentum adversarial Domain Invariant Representation learning (MoDIR), which introduces a momentum method in the DR training process to train a domain classifier on the source versus target, and then adversarially updates the DR encoder to learn domain invariant representations. Our experiments show that MoDIR robustly outperforms its baselines on 10+ ranking datasets from the BEIR benchmark in the zero-shot setup, with more than 10% relative gains on datasets where the evaluation of DR models is sensitive enough. Source code of this paper will be released.
One-sentence Summary: We present a new method that uses mommentum adversarial learning to enforce domain invariant representations and thus improves zero-shot accuracy of dense retrieval.
5 Replies

Loading