A Study on Accessing Linguistic Information in Pre-Trained Language Models by Using Prompts

Marion Di Marco, Katharina Hämmerl, Alexander Fraser

Submission Type: Regular Short Paper

Submission Track: Interpretability, Interactivity, and Analysis of Models for NLP

Submission Track 2: Multilinguality and Linguistic Diversity

Keywords: prompting for linguistic information, morphology, morphological features

TL;DR: We use natural language prompts to formulate linguistic tasks in pre-trained language models.

Abstract: We study whether linguistic information in pre-trained multilingual language models can be accessed by human language: So far, there is no easy method to directly obtain linguistic information and gain insights into the linguistic principles encoded in such models. We use the technique of prompting and formulate linguistic tasks to test the LM’s access to explicit grammatical principles and study how effective this method is at providing access to linguistic features. Our experiments on German, Icelandic and Spanish show that some linguistic properties can in fact be accessed through prompting, whereas others are harder to capture.

Submission Number: 3003