EmbedTextNet: Dimension Reduction with Weighted Reconstruction and Correlation Losses for Efficient Text Embedding

Dae Yon Hwang, Bilal Taha, Yaroslav Nechaev

Abstract: The size of embeddings generated by large lan- guage models can negatively affect system la- tency and model size in certain downstream practical applications (e.g. KNN search). In this work, we propose EmbedTextNet, a light add-on network that can be appended to an ar- bitrary language model to generate a compact embedding without requiring any changes in its architecture or training procedure. Specif- ically, we use a correlation penalty added to the weighted reconstruction loss that better cap- tures the informative features in the text em- beddings, which improves the efficiency of the language models. We evaluated Embed- TextNet on three different downstream tasks: text similarity, language modelling, and text re- trieval. Empirical results on diverse benchmark datasets demonstrate the effectiveness and supe- riority of EmbedTextNet compared to state-of- art methodologies in recent works, especially in extremely low dimensional embedding sizes. The developed code for reproducibility is in- cluded in the supplementary material.