Derivational Probing: Unveiling the Layer-wise Construction of Syntactic Structures in Neural Language Models

Taiga Someya, Ryo Yoshida, Hitomi Yanaka, Yohei Oseki

Keywords: probing, structural probing, syntax, dependency

TL;DR: We introduce *Derivational Probing* to investigate the process by which syntactic structures are built across layer in BERT.

Abstract:

Recent work has demonstrated that neural language models encode syntactic structures in their internal *representations*, yet the *derivations* by which these structures are built across layers remain poorly understood.
In this paper, we introduce *Derivational Probing* to investigate how micro-syntactic structures (e.g., subject phrases) and macro-syntactic structures (e.g., the relationship between the root verb and its direct dependents) are progressively constructed as word embeddings propagate upward across layers.
Our experiments on BERT reveal a clear bottom-up derivation: micro-syntactic dependencies emerge in lower layers and are gradually integrated into a coherent macro-syntactic structure in higher layers.
Furthermore, an analysis on a subject-verb agreement task shows that the timing of macro-structure formation is critical for performance, suggesting an optimal intermediate range for integrating global syntactic information.

Submission Number: 79