Evaluating Gender-Neutral Training Data for Automated Image Captioning

Jack J. Amend, Albatool Wazzan, Richard Souvenir

Abstract: Amassing large-scale datasets used to train machine learning algorithms often includes crowd-sourcing or web scraping. The data resulting from these approaches can carry undesired societal biases that are reflected in the predictions of the learning system. Recently, researchers have proposed mitigation strategies targeting either the learning algorithms or the data used for training. In this paper, we evaluate a simple data augmentation strategy for the task of automated image captioning, namely substituting gendered terms for gender-neutral equivalents. We evaluated this approach on multiple recent image captioning models using both objective and subjective analysis. We found that human raters did not find a difference in the quality of the image captions and, in some cases, the model was able to generate more accurate captions with additional detail when trained with gender-neutral data.