Dataset Fairness: Achievable Fairness On Your Data With Utility Guarantees

Muhammad Faaiz Taufiq, Jean-Francois Ton, Yang Liu

Primary Area: societal considerations including fairness, safety, privacy

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Fairness

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We provide a methodology for reliably characterising the fairness properties of datasets

Abstract: In machine learning fairness, training models which minimize disparity across different sensitive groups often leads to diminished accuracy, a phenomenon known as the fairness-accuracy trade-off. The severity of this trade-off fundamentally depends on dataset characteristics such as dataset imbalances or biases, and therefore using a universal fairness requirement across datasets remains questionable and can often lead to models with varying and substantially low utility. To address this, we present a computationally efficient approach to approximate the fairness-accuracy trade-off curve tailored to individual datasets, backed by rigorous statistical guarantees. By utilizing the You-Only-Train-Once (YOTO) framework, our approach mitigates the computational burden of having to train multiple models when approximating the trade-off curve. Moreover, we introduce confidence intervals around this curve, offering a statistically grounded perspective on acceptable range of fairness violations for any given accuracy threshold. Our empirical evaluation which includes applications to tabular data, computer vision and natural language datasets, underscores that our approach can guide practitioners in accuracy-constrained fairness decisions across various data modalities.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 5969