Generalization through Lexical Abstraction in Transformer Models: The Case of Functional Words

ACL ARR 2025 May Submission3480 Authors

Abstract:

Pronouns, adverbs and other functional words (they, her, somewhere, there) are often used in language to replace concrete nouns or phrases, when their properties -- such as gender, grammatical number -- provide sufficient information for the given context. Do pretrained transformer models encode such functional words in a manner that allows them to be used like humans do? Can language models recognize the syntactic and semantic parallelism of sentences such as "The researchers wrote the paper" and "They wrote it", which relies on such lexical abstraction?

We map these linguistic questions into the embedding space of a pretrained transformer model, and compare representations of nouns, with the representations of the pronouns and adverbs that can replace these nouns, in isolation and in parallel lexicalized and functional sentences. We then probe for shared syntactic and semantic structure in the embeddings of parallel lexicalized and functional sentences.

We find that functional words are located centrally compared to nouns, but are also distinct, which is congruent with their behaviour as place-holders in a wide variety of contexts. The analysis of the embeddings of parallel sentences shows that they do encode the shared syntactic-semantic structure. Moreover, this information is encoded in a similar manner in the representations of functional and lexicalized sentences, thus providing supporting evidence that large language models do encode some form of lexical abstraction.

Paper Type: Long

Research Area: Semantics: Lexical and Sentence-Level

Research Area Keywords: word embeddings, lexical resources, phrase/sentence embedding, word/phrase alignment;

Contribution Types: Model analysis & interpretability, Data resources

Languages Studied: English

Keywords: Functional words, Lexical words, Abstraction, Verb Alternation, English

Submission Number: 3480