FairNNV: The Neural Network Verification Tool For Certifying Fairness

Anne M. Tumlin, Diego Manzanas Lopez, Preston Robinette, Yuying Zhao, Tyler Derr, Taylor T. Johnson

Abstract: Ensuring fairness in machine learning (ML) is vital, especially as these models are increasingly used in socially critical financial decision-making processes such as credit scoring, loan approvals, and fraud detection. Fairness verification aims to provide formal guarantees of fairness in ML models. In this work, we introduce FairNNV, a tool that leverages the Neural Network Verification (NNV) framework to verify individual and counterfactual fairness using reachability analysis techniques. FairNNV introduces the Verified Fairness (VF) score to quantify fairness. Additionally, we compare the verification process of models before and after applying adversarial debiasing techniques to assess the impact of bias mitigation. We demonstrate FairNNV’s effectiveness on several fairness benchmark datasets, including Adult Census, German Credit, and Bank Marketing, with a focused analysis on the impact of adversarial debiasing on Adult Census classifiers. Experimental results show differences between empirical fairness improvements using adversarial debiasing and fairness verification scores with FairNNV, indicating a need for integrating formal verification into the evaluation process to guide model selections when assessing fairness.