Register Always Matters: Analysis of LLM Pretraining Data Through the Lens of Language Variation

Amanda Myntti, Erik Henriksson, Veronika Laippala, Sampo Pyysalo

Keywords: Register, Genre, Large Language Models, NLP, LLM evaluation

TL;DR: We investigate the effect of register (also known as genre) as an explainer of LLM performance, and show that it has a substantial impact on model accuracy on standard benchmarks.

Abstract: Pretraining data curation is a cornerstone in Large Language Model (LLM) development, leading to growing research on quality filtering of large web corpora. From statistical quality flags to LLM-based labelling systems, datasets are divided into categories, frequently reducing to a binary: those passing the filters are deemed as valuable examples, others are discarded as useless or detrimental. However, a more detailed understanding of the contribution of different kinds of texts to model performance is still largely lacking. In this article, we present the first study utilising _registers_ or _genres_—a widely used standard in corpus linguistics to model linguistic variation—to curate pretraining datasets and investigate the effect of register on the performance of LLMs. We train small generative models with register classified data and evaluate them using standard benchmarks, and show that the register of pretraining data substantially affects model performance. We uncover surprising relationships between the pretraining material and the resulting models: using the _News_ register results in subpar performance, and on the contrary, including the _Opinion_ class, covering texts such as reviews and opinion blogs, is highly beneficial. While a model trained on the entire unfiltered dataset outperforms those trained on datasets limited to a single register, combining well-performing registers such as _How-to-Instructions_, _Informational Description_, and _Opinion_ leads to major improvements. Furthermore, analysis of individual benchmark results reveals key differences in the strengths and drawbacks of specific register classes as pretraining data: _How-to-Instructions_ excels at physical reasoning and sentence completion while barely crossing random baselines on world-knowledge benchmarks, while _Narrative_ boosts performance on social interaction tasks but struggles with scientific questions. These findings show that register is an important explainer of model variation and can facilitate more deliberate and detailed future data selection practices.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the COLM Code of Ethics on https://colmweb.org/CoE.html

Author Guide: I certify that this submission complies with the submission instructions as described on https://colmweb.org/AuthorGuide.html

Submission Number: 823