LLM for Zero-Shot Diachronic Semantic Shift Detection

ACL ARR 2025 May Submission3823 Authors

Abstract: This paper explores the use of hidden states from large language models (LLMs) to detect semantic shifts in specialized domains via a zero-shot approach. While encoder-based models dominate this research, they face limitations in context length, computational cost, and interpretability. We propose extracting contextualized word embeddings from the decoder hidden states of Llama 3 series models. Our method employs structured input formulations to guide LLMs in generating context-sensitive word definitions, from which we extract hidden state representations. Using a historical corpus (Credit Suisse Bulletin, 1970--2018), we measure semantic shifts with Jensen-Shannon divergence. Experimental results show decoder hidden states effectively capture contextualized semantics, demonstrated by a case study of the word "interest". To our knowledge, this is the first study leveraging decoder hidden states prompted by definition generation without reliance on generated text analysis. Our method enables decoder-only models to effectively detect semantic shifts, providing a computationally efficient, interpretable alternative for unlabeled data while significantly reducing computational overhead compared to encoder-based approaches.

Paper Type: Long

Research Area: Semantics: Lexical and Sentence-Level

Research Area Keywords: polysemy, lexical semantic change, word embeddings, lexical resources, phrase/sentence embedding

Contribution Types: Model analysis & interpretability, NLP engineering experiment

Languages Studied: English

Submission Number: 3823