Scaling Sentence Embeddings with Large Language Models

Anonymous

Abstract: Large Language Models (LLMs) have recently garnered significant interest. With in-context learning, they achieve impressive results in various natural language tasks. However, the application of LLMs to sentence embeddings remains an area of ongoing research. In this work, we introduce a prompt-based method, PromptEOL, designed to improve LLMs performance on sentence embeddings with explicit one word limitation. We further integrate in-context learning to refine sentence embeddings. Our extensive experiments demonstrate that in-context learning allows LLMs to generate superior sentence embeddings without any fine-tuning, enabling them to perform comparably to current contrastive learning methods. We also investigate the integration of contrastive learning with PromptEOL. Notably, the 2.7B OPT model, when combined our method, surpasses the previous state-of-the-art method with 4.8B parameters. In addition, we propose a novel method based on Direct Performance Optimization (DPO) to better align the embeddings. With our methods, we successfully achieve an 86.76 Spearman correlation on STS tasks, a 1.8 improvement over the previous methods.

Paper Type: long

Research Area: Semantics: Sentence-level Semantics, Textual Inference and Other areas

Languages Studied: English