NoCoLA: The Norwegian Corpus of Linguistic Acceptability

Matias Jentoft, David Samuel

Keywords: nocola, linguistic acceptability, language models, nlp, norwegian, evaluation, ask

TL;DR: NoCoLa: datasets for evaluating the linguistic knowledge of Norwegian language models

Abstract: While there has been a surge of large language models for Norwegian in recent years, we lack any tool to evaluate their understanding of grammaticality. We present two new Norwegian datasets for this task. NoCoLA-class is a supervised binary classification task where the goal is to discriminate between acceptable and non-acceptable sentences. On the other hand, NoCoLA-zero is a purely diagnostic task for evaluating the grammatical judgement of a language model in a completely zero-shot manner, i.e. without any further training. In this paper, we describe both datasets in detail, show how to use them for different flavors of language models, and conduct a comparative study of the existing Norwegian language models.

Student Paper: Yes, the first author is a student

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:2306.07790/code)