Advancing Equitable AI: A Comprehensive Framework for Individual Fairness Assessment

Yichen Wang, Yaxuan Li, Wenjie Zhu

Keywords: Social Welfare, Justice, Fairness and Equality, Philosophical and Ethical Issues

Abstract: Ensuring fairness in machine learning (ML) models is essential for developing equitable and trustworthy AI systems. There has been extensive existing research on group-based fairness metrics such as the Statistical Parity Difference and Disparate Impact, but these group-based fairness metrics often fail to address fairness at the individual level. An ML model can achieve perfect group fairness, but produce discriminatory outcomes at the individual level or vice versa. In this paper, four novel individual-based fairness metrics are proposed: Proxy Dependency Score, Counterfactual Stability Rate, Attributional Independence Score, and Intra-Cohort Decision Consistency. These metrics are designed to assess different facets of individual fairness, including protected attributes’ influence on model predictions, model’s robustness to protected attribute perturbations, the independence of attributions from protected attributes, and the consistency within similar individuals. These four new individual-based metrics are empirically compared with group outcome-based fairness metrics on ML models trained on Adult and COMPAS datasets. The empirical results reveal that models deemed unfair by group metrics may exhibit individual-level fairness. Our work highlights the critical need for comprehensive individual fairness assessments in real-world applications. Our proposed framework can act as a complement to group-based evaluations towards a more complete understanding of AI fairness and the development of more equitable AI systems.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 1048