DRAMA: Diverse Augmentation from Large Language Models Towards Smaller Generalizable Dense Retrievers

ACL ARR 2025 February Submission2562 Authors

Abstract:

Large language models (LLMs) have demonstrated strong effectiveness and robustness when fine-tuned as dense retrievers.
However, their large parameter size presents significant computational challenges at inference time.
While smaller retrievers offer better efficiency, they often fail to generalize effectively with limited supervised fine-tuning data.
In this work, we introduce DRAMA, a training framework that leverages LLMs to train smaller generalizable dense retrievers.
In particular, we adopt pruned LLMs as the backbone and train on diverse LLM-augmented data in a single-stage contrastive learning setup.
Experiments show that DRAMA offers better multilingual and long-context capabilities than traditional encoder-based retrievers, and achieves strong effectiveness across multiple tasks and languages.

Paper Type: Long

Research Area: Information Retrieval and Text Mining

Research Area Keywords: dense retrieval, large language model, data augmentation

Contribution Types: NLP engineering experiment

Languages Studied: English, Arabic, Bengali, Spanish, Persian, Finnish, French, Hindi, Indonesian, Japanese, Korean, Russian, Swahili, Telugu, Thai, Chinese, German, Yoruba, Italian, Portuguese.

Submission Number: 2562