Benchmarking Bias Mitigation Algorithms in Representation Learning through Fairness Metrics

Charan Reddy, Soroush Mehri, Deepak Sharma, Samira Shabanian, Sina Honari

Keywords: fairness model evaluation, fair deep learning, adversarial fairness

Abstract: With the recent expanding attention of machine learning researchers and practitioners to fairness, there is a void of a common framework to analyze and compare the capabilities of proposed models in deep representation learning. In this paper, we evaluate different fairness methods trained with deep neural networks on a common synthetic dataset to obtain a better insight into the working of these methods. In particular, we train about 2000 different models in various setups, including unbalanced and correlated data configurations, to verify the limits of the current models and better understand in which setups they are subject to failure. In doing so we present a dataset, a large subset of proposed fairness metrics in the literature, and rigorously evaluate recent promising debiasing algorithms in a common framework hoping the research community would take this benchmark as a common entry point for fair deep learning.

One-sentence Summary: Benchmarking Bias Mitigation Algorithms in Representation Learning through Fairness Metrics

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Reviewed Version (pdf): https://openreview.net/references/pdf?id=5FdcE9Dm9