Preliminary Experiments to Examine the Stability of Bias-Aware Techniques

Toshihiro Kamishima, Shotaro Akaho, Yukino Baba, Hisashi Kashima

Abstract: Fairness-aware techniques are designed to remove socially sensitive information, such as gender or race. Many types of fairness-aware predictors have been developed, but they were designed essentially to improve the accuracy or fairness of the prediction results. We focus herein on another aspect of fairness-aware predictors, i.e., the stability. We define that fairness-aware techniques are stable if the same models are learned when a training dataset contains the same information except for the sensitive information. We sought to collect benchmark datasets to investigate such stability. We collected preference data in a manner ensuring that the users’ responses were influenced by cognitive biases. If the same models are learned for a dataset influenced by different types of cognitive biases, the learner of the models can be considered stable. We performed preliminary experiments using this dataset, but we failed to fully remove the influence of cognitive biases. We discuss the necessary next steps to solve this problem.