DADR: A Denoising Approach for Dense Retrieval Model Training

Mengxue Du, Shasha Li, Jie Yu, Jun Ma, Huijun Liu, Miaomiao Li, Bin Ji

Abstract: With the development of representation learning techniques, Dense Retrieval (DR) has become a new paradigm to retrieve relevant texts for better ranking performance. Although current DR models have achieved encouraging results, their performance is highly affected by the noise level in training samples. In particular, a large number of examples that were not labeled as positives (which were used as negative samples by default) were found to actually be positive or highly relevant. As such, it is of critical importance to account for the inevitable noises in DR model training. However, little work on dense retrieval has taken the noisy nature into consideration. In this work, we intensely investigate the serious negative impacts of noisy training samples and propose a new denoising approach, i.e., A Denoising Approach based on dynamic weights for Dense Retrieval model training (DADR), which reduces the effects of noise on model performance by assigning diverse weights to the different samples during the training process. We incorporate the proposed DADR approach with three representative kinds of sampling methods and different loss functions. Experimental results on two publicly available retrieval benchmark datasets show that our approach significantly improves the performance of the DR model over normal training.