Speeding up fairness reductions

Abstract: We study the problem of fair classification, where the goal is to optimize classification accuracy subject to fairness constraints. This type of problem occurs in many real-world applications, where we seek to assure that a deployed AI system does not disproportionally impact historically disadvantaged groups. One of the leading approaches in the literature is the reduction approach (Agarwal et al., 2018; 2019), which enjoys many favorable properties. For instance, it supports a wide range of fairness constraints and model families and is usually easy to incorporate in existing ML pipelines. The reduction approach acts as a wrapper around a standard ML algorithm and obtains a model that satisfies fairness constraints by repeatedly running a fairness-unaware base algorithm. A typical number of iterations is around 100, meaning that the reduction approach can be up to 100 times slower than the base algorithm, which limits its applicability. To overcome this limitation, we introduce two algorithmic innovations. First, we interleave the exponentiated gradient updates of the standard reduction approach with column-generation updates, which leads to a decrease in the number of calls to the base algorithm. Second, we introduce adaptive sampling, which decreases the sizes of the datasets used in the calls to the base algorithm. We conduct comprehensive experiments to evaluate efficacy of our improvements, showing that our two innovations speed up the reduction approach by an order of magnitude without sacrificing the quality of the resulting solutions.

Submission Type: Long submission (more than 12 pages of main content)

Changes Since Last Submission:

2nd round of changes (on 4/10, marked in orange):
* Added additional tradeoff points for ZAFAR DI (see Figure 4a and the caption for Figure 4 in Appendix A)
* Added new experiments with FairGBM (see Figure 5b and the caption for Figure 5 in Appendix A)
* Corresponding writing updates at the end of Section 5.1 (paragraph "Models"), and the second paragraph of Section 5.2
* Clarified that it is just the implementations of ZAFAR DI/EO that are limited to logistic regression (Section 5.1, paragraph "Models")

1st round of changes (on 4/6, marked in blue except for new entries in the reference section and the addition of two new citations in paragraph 2 of Section 2):
* Added "Usage guidelines, risks, and limitations" section (Section 1.1) at the end of Section 1
* Added (Cruz et al., 2023) and (Cruz & Hardt, 2024) in the discussion of related work, paragraph 2 of Section 2
* Added discussion of AutoML methods at the end of Section 2
* Added formal justification for the adaptive sampling method in Appendix B, and a pointer to it in the last paragraph of Section 4
* Added explanation that baselines were just evaluated at their default hyperparameters and that their goal is just to provide context for the metric values (end of Section 5.1, and also edits in the second paragraph of Section 5.2 and caption of Figure 1)

Assigned Action Editor: ~Shahin_Jabbari1

Submission Number: 7418