RuBia: A Russian Language Bias Detection Dataset

Anonymous

Abstract: Warning: this paper contains content that may be offensive or upsetting.Pre-trained language models are often affected by the social and cultural biases present in the training data. To test if a model’s behavior is fair, functional challenge datasets are developed. However, a limited number of such datasets exists, and the included data are mostly limited to sentences in English depicting US cultural stereotypes. In this paper, we propose \texttt{RuBia}: a bias detection dataset for the Russian Language. The data in the dataset are divided into 4 domains, each corresponding to a specific way a bias or prejudice can be reflected in the language. Each example in the dataset consists of two sentences where the first reinforces a potentially harmful stereotype or trope while the second contradicts it. Overall, there are 2561 sentence pairs, organized into 19 fine-grained subdomains.To illustrate RuBia's purpose, we conduct diagnostic evaluation of six near-state-of-the-art Transformer-based language models and discuss models predespostition to social biases. Our pipeline to data collection is easy to reproduce and extend to other languages and cultures. We release the code, developed to collect the data and score the models, in open access. The url will be available in the final version of the paper.

Paper Type: long