Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction

Taeuk Kim, Jihun Choi, Daniel Edmiston, Sang-goo Lee

Abstract:

With the recent success and popularity of pre-trained language models (LMs) in natural language processing, there has been a rise in efforts to understand their inner workings.
In line with such interest, we propose a novel method that assists us in investigating the extent to which pre-trained LMs capture the syntactic notion of constituency.
Our method provides an effective way of extracting constituency trees from the pre-trained LMs without training.
In addition, we report intriguing findings in the induced trees, including the fact that pre-trained LMs outperform other approaches in correctly demarcating adverb phrases in sentences.

Code: [ galsang/trees_from_transformers](https://github.com/galsang/trees_from_transformers)

Data: [MultiNLI](https://paperswithcode.com/dataset/multinli)

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2002.00737/code)