Isotropic Representation Can Improve Dense Retrieval

Euna Jung, Jungwon Park, Jaekeol Choi, Sungyoon Kim, Wonjong Rhee

Abstract: The latest Dense Retrieval (DR) models typically encode queries and documents using BERT and subsequently apply a cosine similarity-based scoring to determine the relevance. BERT representations, however, are known to follow an anisotropic distribution of a narrow cone shape and such an anisotropic distribution can be undesirable for relevance estimation. In this work, we first show that BERT representations in DR also follow an anisotropic distribution. We adopt unsupervised post-processing methods of Normalizing Flow and whitening to cope with the problem, and develop a token-wise method in addition to the sequence-wise method. We show that the proposed methods can effectively enhance the isotropy of representations, thereby improving the performance of DR models such as ColBERT and RepBERT. To examine the potential of isotropic representation for improving the robustness of DR models, we investigate out-of-distribution tasks where the test dataset differs from the training dataset. The results show that isotropic representation can certainly achieve a generally improved performance (The code is available at https://github.com/SNU-DRL/IsotropicIR.git ).