AccSent: Accurate Semantic Evaluation of Sentence Embeddings

Anonymous

TL;DR: We present a new sentence-level semantic evaluation set.

Abstract: Sentence embeddings, which encode arbitrary sentences as fixed-length numeric vectors, have shown promising results across a diverse range of semantic tasks. While prior work has demonstrated their effectiveness at capturing basic semantics, we present a new semantic evaluation set called AccSent for a more in-depth analysis of how accurately such embeddings reflect sentence semantics. We show that current embedding models are generally able to capture the broad semantic meaning of sentences, but that they are heavily affected by surface-level biases (such as lexical choices and sentence structures) instead of capturing the accurate semantic meaning of sentences. With our AccSent test set, sentence embedding models can only obtain an accuracy of 26.2% for evaluating the semantic similarity of sentences. We release our data and code on GitHub.

Paper Type: short

Research Area: Semantics: Sentence-level Semantics, Textual Inference and Other areas

Contribution Types: Model analysis & interpretability, Data resources

Languages Studied: English