The cosine measure is too shallow: Evidence from the sentence embedding space

ACL ARR 2026 January Submission3477 Authors

Keywords: sentence embeddings, embedding space, foundation models, cosine

Abstract:

Embedding spaces allow us to manipulate words and sentences as mathematical objects, through various metrics as proxies of linguistic properties, mainly similarity or relatedness. Most metrics treat each dimension separately. We present an investigation of the sentence embedding space of several foundation models, that show that distance metrics that consider each embedding dimension separately are too shallow to capture the complexity of similarity between embeddings that encode multiple layers of linguistic information.

To avoid using subjective similarity scores, we compare three sentence representation variations from each model. We explore whether the distance between these sentence embedding variations is reflected in their relative performance on linguistic tasks, and whether they encode the same information in a similar manner. The results show that cosine similarity in this space is not predictive of the representations' relative performances on a variety of tasks in the HOLMES dataset. Furthermore, by allowing that linguistic information -- in particular chunk structure -- is encoded as weighted combinations of features, we detect this information, encoded the same way, in all three representation variations.

Transformer models are deep. Our analysis shows that the embeddings they induce are complex objects with internal structure, and are not simply points in a space, or shallow arrays of numbers.

Paper Type: Long

Research Area: Interpretability and Analysis of Models for NLP

Research Area Keywords: knowledge tracing/discovering/inducing, probing, feature attribution

Contribution Types: Model analysis & interpretability

Languages Studied: English, (French, Italian, Spanish, German, Romanian on part of the experiments)

Submission Number: 3477