What do you learn from context? Probing for sentence structure in contextualized word representationsDownload PDF

27 Sept 2018, 22:38 (edited 10 Feb 2022)ICLR 2019 Conference Blind SubmissionReaders: Everyone
  • Keywords: natural language processing, word embeddings, transfer learning, interpretability
  • TL;DR: We probe for sentence structure in ELMo and related contextual embedding models. We find existing models efficiently encode syntax and show evidence of long-range dependencies, but only offer small improvements on semantic tasks.
  • Abstract: Contextualized representation models such as ELMo (Peters et al., 2018a) and BERT (Devlin et al., 2018) have recently achieved state-of-the-art results on a diverse array of downstream NLP tasks. Building on recent token-level probing work, we introduce a novel edge probing task design and construct a broad suite of sub-sentence tasks derived from the traditional structured NLP pipeline. We probe word-level contextual representations from four recent models and investigate how they encode sentence structure across a range of syntactic, semantic, local, and long-range phenomena. We find that existing models trained on language modeling and translation produce strong representations for syntactic phenomena, but only offer comparably small improvements on semantic tasks over a non-contextual baseline.
  • Code: [![Papers with Code](/images/pwc_icon.svg) 2 community implementations](https://paperswithcode.com/paper/?openreview=SJzSgnRcKX)
  • Data: [Billion Word Benchmark](https://paperswithcode.com/dataset/billion-word-benchmark)
9 Replies

Loading