A Comprehensive Study of Gender Bias in Chemical Named Entity Recognition Models

Anonymous

Abstract: Chemical named entity recognition (NER) models influence numerous downstream tasks, from adverse drug reaction identification to pharmacoepidemiology. However, it is unknown whether these models work the same for everyone. Performance disparities can potentially cause harm rather than the intended good. This paper assesses gender-related performance disparities in chemical NER systems. We develop a framework for measuring gender bias in chemical NER models using synthetic data and a newly annotated corpus of over 92,405 words with self-identified gender information from Reddit. Our evaluation of state-of-the-art biomedical NER models reveals evident biases. For instance, synthetic data suggests female-related names are frequently misclassified as chemicals, especially with datasets rich in brand names. Additionally, we observe significant performance disparities between female- and male-associated data in both datasets. Many systems fail to detect contraceptives such as birth control. Our findings emphasize the biases in chemical NER models, urging practitioners to be aware of and address these biases in application.

Paper Type: long

Research Area: Computational Social Science and Cultural Analytics

Languages Studied: English