Multiaccuracy and Multicalibration via Proxy Groups

Beepul Bharti, Mary Versa Clemens-Sewall, Paul Yi, Jeremias Sulam

TL;DR: This work demonstrates how to evaluate and control for multiaccuracy and multicalibration when sensitive group data is missing by using proxies.

Abstract: As the use of predictive machine learning algorithms increases in high-stakes decision-making, it is imperative that these algorithms are fair across sensitive groups. However, measuring and enforcing fairness in real-world applications can be challenging due to missing or incomplete sensitive group information. Proxy-sensitive attributes have been proposed as a practical and effective solution in these settings, but only for parity-based fairness notions. Knowing how to evaluate and control for fairness with missing sensitive group data for newer, different, and more flexible frameworks, such as multiaccuracy and multicalibration, remains unexplored. In this work, we address this gap by demonstrating that in the absence of sensitive group data, proxy-sensitive attributes can provably be used to derive actionable upper bounds on the true multiaccuracy and multicalibration violations, providing insights into a predictive model’s potential worst-case fairness violations. Additionally, we show that adjusting models to satisfy multiaccuracy and multicalibration across proxy-sensitive attributes can significantly mitigate these violations for the true, but unknown, sensitive groups. Through several experiments on real-world datasets, we illustrate that approximate multiaccuracy and multicalibration can be achieved even when sensitive group data is incomplete or unavailable.

Lay Summary: As machine learning models are increasingly used to make important decisions, like who gets a loan or access to healthcare, it’s critical that they treat different sensitive groups fairly. In many real-world cases, however, we don’t have complete information about people’s sensitive attributes, like race or biological sex. This makes it hard to evaluate or enforce fairness. In this work, we show that we can still make fairness assessments using proxy sensitive attributes. While proxy attributes have been used before for traditional notions of fairness, we extend their use to newer, more flexible fairness concepts called multiaccuracy and multicalibration. We prove that proxy attributes can help estimate how unfair a model might be in the worst case, even without knowing the true sensitive groups. We also show that building models to satisfy fairness across these proxies can lead to useful guarantees on a model's fairness. We test our approach on real datasets to illustrate the utility of our results.

Link To Code: https://github.com/Sulam-Group/proxy_ma-mc

Primary Area: Social Aspects->Fairness

Keywords: multicalibration, multiaccuracy, fairness

Submission Number: 12229