The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual DatasetDownload PDF

06 Jun 2022, 09:42 (modified: 31 Oct 2022, 16:14)NeurIPS 2022 Datasets and Benchmarks Readers: Everyone
Keywords: BigScience, Dataset, Multilingual, Language Modeling
TL;DR: 1.6TB multilingual dataset created collaboratively within BigScience to train language models
Abstract: As language models grow ever larger, the need for large-scale high-quality text datasets has never been more pressing, especially in multilingual settings. The BigScience workshop, a 1-year international and multidisciplinary initiative, was formed with the goal of researching and training large language models as a values-driven undertaking, putting issues of ethics, harm, and governance in the foreground. This paper documents the data creation and curation efforts undertaken by BigScience to assemble the Responsible Open-science Open-collaboration Text Sources (ROOTS) corpus, a 1.6TB dataset spanning 59 languages that was used to train the 176-billion-parameter BigScience Large Open-science Open-access Multilingual (BLOOM) language model. We further release a large initial subset of the corpus and analyses thereof, and hope to empower large-scale monolingual and multilingual modeling projects with both the data and the processing tools, as well as stimulate research around this large multilingual corpus.
Supplementary Material: pdf
Dataset Url: Data: Tooling:
License: - Each constituent subset of the dataset will be released under the license that applies to it. (See individual dataset page for specific license information: - Tooling code released under Apache 2.0
Author Statement: Yes
Contribution Process Agreement: Yes
In Person Attendance: Yes
17 Replies