Algorithmic fairness with monotone likelihood ratios

TMLR Paper4684 Authors

Abstract: We show that inequalities of many commonly used fairness metrics (true/false positive/negative rates, predicted positive/negative rates, and positive/negative predictive values) are guaranteed for groups with different outcome rates under a monotonically calibrated model whose risk distributions have a monotone likelihood ratio, extending existing impossibility results. We further provide lower bounds on the FNR/FPR disparities and PPR/PNR disparities in the same setting, showing that either the FNR disparity or FPR disparity is at least as large as the positive outcome rate disparity (for FNR disparity) or negative outcome rate disparity (for FPR disparity), and either the PPR disparity or PNR disparity is at least as large as the positive outcome rate disparity (for PPR disparity) or negative outcome rate disparity (for PNR disparity). While incompatibilities of some combinations of these metrics have been demonstrated previously, we are unaware of any work that has demonstrated direct incompatibility of calibration with these individual equalities, equivalence of these inequalities, or lower bounds for the disparity in these values under distributional assumptions about a model's predictions.

Submission Length: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission:

We have made edits to address comments made by the reviewers, primarily:
1) Expanding the discussion of empirical evidence for the MLRP assumption
2) Application of the results to another case study
3) Addition of text between lemmas to improve readability
4) Addition of missing definitions for certain metrics
Changed/added sections are noted in blue.

Assigned Action Editor: ~Geoff_Pleiss1

Submission Number: 4684