Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar InductionDownload PDF

25 Sept 2019, 19:15 (modified: 23 Jan 2023, 18:07)ICLR 2020 Conference Blind SubmissionReaders: Everyone
Original Pdf: pdf
Code: [![github](/images/github_icon.svg) galsang/trees_from_transformers](
Data: [MultiNLI](, [Penn Treebank](
Abstract: With the recent success and popularity of pre-trained language models (LMs) in natural language processing, there has been a rise in efforts to understand their inner workings. In line with such interest, we propose a novel method that assists us in investigating the extent to which pre-trained LMs capture the syntactic notion of constituency. Our method provides an effective way of extracting constituency trees from the pre-trained LMs without training. In addition, we report intriguing findings in the induced trees, including the fact that pre-trained LMs outperform other approaches in correctly demarcating adverb phrases in sentences.
10 Replies