Structure and Component: Factors Influencing the Compositionality Development of Pre-trained Language Models During Fine-tuning

ACL ARR 2026 January Submission2984 Authors

Keywords: compositionality, structure, component, fine-tuning, pre-trained language model

Abstract: In this work, we investigate factors influencing the compositionality development of pre-trained language models during fine-tuning from the perspective of two key concepts in linguistic compositionality: structure and component. Our key finding is that even fine-tuning of identical symbolic essences can lead to different compositionality developments of the same pre-trained language model, driven by the following factors: (1) structures uncommon in natural language lead to high compositionality, (2) components uncommon in natural language lead to high compositionality, and (3) component combinations uncommon in natural language lead to low compositionality. This phenomenon is intrinsically linked to linguistic compositionality, offering a new perspective for compositionality research on language models.

Paper Type: Short

Research Area: Semantics: Lexical, Sentence-level Semantics, Textual Inference and Other areas

Research Area Keywords: compositionality

Contribution Types: NLP engineering experiment

Languages Studied: English

Submission Number: 2984