Derivational Probing: Unveiling the Layer-wise Derivation of Syntactic Structures in Neural Language Models

Taiga Someya, Ryo Yoshida, Hitomi Yanaka, Yohei Oseki

Keywords: probing, structural probing, syntax, dependency

TL;DR: We introduce *Derivational Probing* to investigate the process by which syntactic structures are built across layer in BERT.

Abstract:

Recent work has demonstrated that neural language models encode syntactic structures in their internal *representations*, yet the *derivations* by which these structures are constructed across layers remain poorly understood.
In this paper, we propose *Derivational Probing* to investigate how micro-syntactic structures (e.g., subject noun phrases) and macro-syntactic structures (e.g., the relationship between the root verbs and their direct dependents) are constructed as word embeddings propagate upward across layers.
Our experiments on BERT reveal a clear bottom-up derivation: micro-syntactic structures emerge in lower layers and are gradually integrated into a coherent macro-syntactic structure in higher layers.
Furthermore, a targeted evaluation on subject-verb number agreement shows that the timing of constructing macro-syntactic structures is critical for downstream performance, suggesting an optimal timing for integrating global syntactic information.

Submission Number: 79