Leveraging LLMs for Synthesizing Training Data Across Many Languages in Multilingual Dense Retrieval

Anonymous

TL;DR: JUMP-IR is a synthetic retrieval training dataset containing 33 languages for fine-tuning multilingual dense retrievers without requiring any human supervision.

Abstract: There has been limited success for dense retrieval models in multilingual retrieval, due to uneven and scarce training data available across multiple languages. Synthetic training data generation is promising (e.g., InPars or Promptagator), but has been investigated only for English. Therefore, to study model capabilities across both cross-lingual and monolingual retrieval tasks, we develop JUMP-IR, a synthetic retrieval training dataset containing 33 (high to very-low resource) languages for fine-tuning multilingual dense retrievers without requiring any human supervision. To construct JUMP-IR, we propose SAP (summarize-then-ask prompting), where the large language model (LLM) generates a textual summary prior to the query generation step. SAP assists the LLM in generating informative queries in the target language. Using JUMP-IR, we explore synthetic fine-tuning of multilingual dense retrieval models and evaluate them robustly on three retrieval benchmarks: XOR-Retrieve (cross-lingual), XTREME-UP (cross-lingual), and MIRACL (monolingual). Our models, called JUMP-X, are competitive with human-supervised dense retrieval models, e.g., mContriever, finding that JUMP-IR can cheaply substitute for expensive human-labeled retrieval training data.

Paper Type: long

Research Area: Information Retrieval and Text Mining

Contribution Types: Approaches to low-resource settings, Approaches low compute settings-efficiency, Data resources

Languages Studied: Arabic (ar), Assamese (as), Bengali (bn), Bhojpuri (bho), German (de), English (en), Spanish (es), Persian (fa), Finnish (fi), French (fr), Konkani (gom), Gujarati (gu), Hindi (hi), Indonesian (id), Japanese (ja), Kannada (kn), Korean (ko), Maithili (mai), Malayalam (ml), Manipuri (mni), Marathi (mr), Odia (or), Punjabi (pa), Pashto (ps), Russian (ru), Sanskrit (sa), Swahili (sw), Tamil (ta), Thai (th), Urdu (ur), Yoruba (yo), Chinese (zh)