Using Artificial French Data to Understand the Emergence of Gender Bias in Transformer Language Models
Submission Type: Regular Short Paper
Submission Track: Resources and Evaluation
Submission Track 2: Language Modeling and Analysis of Language Models
Keywords: Language Model, Transformer, LM Analysis, Gender Bias
TL;DR: by learning language models on artificial corpora, highlight the conditions under which linguistic properties such as gender can emerge; our experiments show in particular that gender bias can appear even in perfectly gender-balanced corpora
Abstract: Numerous studies have demonstrated the ability of neural language models to learn various linguistic properties without direct supervision. This work takes an initial step towards exploring the less researched topic of how neural models discover linguistic properties of words, such as gender, as well as the rules governing their usage. We propose to use an artificial corpus generated by a PCFG based on French to precisely control the gender distribution in the training data and determine under which conditions a model correctly captures gender information or, on the contrary, appears gender-biased.
Submission Number: 4552
Loading