Fairness Metric Impossibility: Investigating and Addressing Conflicts

Jake Robertson, Noor Awad, Thorsten L Schmidt, Frank Hutter

Primary Area: societal considerations including fairness, safety, privacy

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Fairness, Multi-objective optimization, Hyperparameter Optimization

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract: Fairness-aware ML (FairML) applications are often characterized by intricate social objectives and legal requirements, often encompassing multiple, potentially conflicting notions of fairness. Despite the well-known Impossibility Theorem of Fairness and vast theoretical research on the statistical and socio-technical trade-offs between fairness metrics, many FairML approaches still optimize for a single, user-defined fairness objective. However, this one-sided optimization can inadvertently lead to violations of other pertinent notions of fairness, resulting in adverse social consequences. In this exploratory and empirical study, we address the presence of fairness-metric conflicts by treating fairness metrics as conflicting objectives in a multi-objective (MO) sense. To efficiently explore multiple fairness-accuracy trade-offs and effectively balance conflicts between various fairness objectives, we introduce the ManyFairHPO framework, a novel many-objective (MaO) hyper-parameter optimization (HPO) approach. By enabling fairness practitioners to specify and explore complex and multiple fairness objectives, we open the door to further socio-technical research on effectively combining the complementary benefits of different notions of fairness.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 1191