MKD: Mutual Knowledge Distillation for Membership Privacy Protection

Sihao Huang, Zhongxiang Liu, Jiafu Yu, Yongde Tang, Zidan Luo, Yuan Rao

Abstract: Machine learning models are susceptible to member inference attacks, which attempt to determine whether a given sample belongs to the training data set of the target model. The significant privacy concerns raised by member inference have led to the development of various defenses against Member Inference Attacks (MIAs). Existing techniques for knowledge distillation have been identified as a potential solution to mitigate the tradeoff between model performance and data privacy, demonstrating promising results. Nonetheless, the limitations in performance imposed by the teacher model in knowledge distillation, along with the scarcity of unlabeled reference data, present a challenge in achieving high-performance privacy-preserving training for the target model. To address these issues, we propose a novel knowledge distillation based defense method, i.e., Mutual Knowledge Distillation (MKD). Dividing the training set into subsets for the teacher and the student models, MKD trains them through mutual knowledge distillation for mitigating MIAs. Extensive experimental results demonstrate that MKD outperforms several existing defense methods in improving the trade-off between model utility and member privacy.