A Study on Accessing Linguistic Information in Pre-Trained Language Models by Using Prompts

Marion Di Marco, Katharina Hämmerl, Alexander Fraser

Abstract: We study whether linguistic information in pre- trained multilingual language models can be accessed by human language: So far, there is no easy method to directly obtain linguistic in- formation and gain insights into the linguistic principles encoded in such models. We use the technique of prompting and formulate linguis- tic tasks to test the LM’s access to explicit gram- matical principles and study how effective this method is at providing access to linguistic fea- tures. Our experiments on German, Icelandic and Spanish show that some linguistic proper- ties can in fact be accessed through prompting, whereas others are harder to capture.