From Attack to Restoration: A Two-Stage Diffusion Framework for Face Privacy

Xuanyang Li, Shiwei Li, Jianxiang Liao, Wang Yu-Hang, Jian Liu, Wenfei Yin

Keywords: Face privacy protection, Diffusion models

TL;DR: Two-stage diffusion with Negative Prompt Inversion and angular-margin loss delivers state-of-the-art face-privacy protection while keeping images natural.

Abstract: The surge of facial photos on social media has made unauthorized face recognition (FR) a serious threat to personal privacy. Existing diffusion-based privacy methods are vulnerable to the purification effect, which weakens adversarial signals, and their single-stage optimization struggles to balance deceptiveness and visual quality. To address this, we propose a two-stage face privacy protection framework. In Stage 1, we introduce Negative Prompt Inversion (NPI) into the diffusion reverse process and incorporate an angular margin constraint to steer features toward a target identity in feature space—counteracting the dilution of adversarial signals at the source and mitigating gradient conflicts and trade-off issues. Stage 2 focuses on perceptual quality, using perceptual loss and regularization strategies to enhance naturalness while preserving the method's ability to deceive recognizers. Extensive experiments on the CelebA-HQ and LADN public datasets show that our approach achieves state-of-the-art protection success rates (PSR) while maintaining high image quality, underscoring its promise for privacy protection and real-world deployment.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 3390