Analysing the Linearity of Linguistic Relations in Language Model Embedding Spaces

Vasudevan Nedumpozhimana, Fathima Thekkekara, John Kelleher

Keywords: Representational Analysis, Linear Representational Hypothesis, Linguistic Relations, Language Models

TL;DR: The paper propose a novel framework for analysing linearity of linguistic relations in the representational space of language models and present preliminary empirical analysis.

Abstract: We propose a framework to analyse how strongly different linguistic relations are linearly encoded in language model embedding spaces. We formalise linear encoding via a constrained linear approximation over related and unrelated word pairs and apply this to an extended BATS dataset covering inflectional, derivational, lexicographic, and encyclopedic relations in GloVe, RoBERTa, and ModernBERT. Our experiments show near-perfect linear encodings for inflectional and derivational relations, but substantially higher errors for lexicographic and encyclopedic relations, especially for one-to-many and many-to-many associations. We also find that RoBERTa and ModernBERT generally encode relations more linearly than GloVe. These results indicate that our framework can reveal which relational structures are most linearly accessible in embeddings, offering a compact tool for probing and comparing relational geometry across models.

Anonymization: This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs.

Style Files: I have used the style files.

Submission Number: 13