Linear Relational Decoding of Morphological Relations in Language Models
Keywords: LMs, Model Interpretability, Representation Theory, Morphology, Linear Representation Hypothesis, Analogical Reasoning
TL;DR: In LMs, we show that the linear approximation Ws, where s is an early hidden state and W is a local derivative, can approximate morphology highly accurately, suggesting linear relational embeddings form a key aspect of their linguistic fluency.
Abstract: The recent success of transformer language models owes much to their conversational fluency and productivity in linguistic and morphological aspects. An affine Taylor approximation has been found to be a good approximation for transformer computations over certain factual and encyclopedic relations. We show that the truly linear approximation $W\textbf{s}$, where $\textbf{s}$ is a middle layer representation of the base form and $W$ is a local model derivative, is necessary and sufficient to approximate morphological derivations. This approach achieves above 80\% faithfulness across most morphological tasks in the Bigger Analogy Test Set, and is successful across language models and typological categories. We propose that morphological relationships in transformer models are likely to be linearly encoded, with implications for how entities are represented in latent space.
Primary Area: foundation or frontier models, including LLMs
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 12211
Loading