Translate-distill: Learning cross-language dense retrieval by translation and distillation

Eugene Yang, Dawn Lawrie, James Mayfield, Douglas W. Oard, Scott Miller

Abstract:

Prior work on English monolingual retrieval has shown that a crossencoder trained using a large number of relevance judgments for query-document
pairs can be used as a teacher to train more efficient, but similarly effective, dualencoder student models. Applying a similar knowledge distillation approach to
training an efficient dual-encoder model for Cross-Language Information Retrieval
(CLIR), where queries and documents are in different languages, is challenging
due to the lack of a sufficiently large training collection when the query and
document languages differ. The state of the art for CLIR thus relies on translating
queries, documents, or both from the large English MS MARCO training set, an
approach called Translate-Train. This paper proposes an alternative, TranslateDistill, in which knowledge distillation from either a monolingual cross-encoder
or a CLIR cross-encoder is used to train a dual-encoder CLIR student model. This
richer design space enables the teacher model to perform inference in an optimized
setting, while training the student model directly for CLIR. Trained models and
artifacts are publicly available on Huggingface.