DERD: Data-free Adversarial Robustness Distillation through Self-adversarial Teacher Group

Yuhang Zhou, Yushu Zhang, Leo Yu Zhang, Zhongyun Hua

Abstract:

Computer vision models based on deep neural networks are proven to be vulnerable to adversarial attacks. Robustness distillation, as a countermeasure, takes both robustness challenges and efficiency challenges of edge models into consideration. However, most existing robustness distillations are data-driven, which can hardly be deployed in the data-privacy scenarios. Also, the trade-off between robustness and accuracy tends to transfer from the teacher to the student along with the robustness, and there has been no discussion on mitigating this trade-off in the data-free scenario yet.
In this paper, we propose a Data-free Experts-guided Robustness Distillation (DERD) to extend robustness distillation to data-free paradigm, which offers three advantages:
(1) Dual-level adversarial learning strategy achieves robustness distillation without the real data.
(2) Expert-guided distillation strategy brings a better trade-off to the student model.
(3) A novel stochastic gradient aggregation module reconciles the task conflicts of the multi-teacher from a consistency perspective. Extensive experiments demonstrate that the proposed DERD can even achieve comparable results to data-driven methods.

Primary Subject Area: [Experience] Multimedia Applications

Secondary Subject Area: [Generation] Social Aspects of Generative AI, [Content] Vision and Language, [Generation] Multimedia Foundation Models

Relevance To Conference: In the context of data privacy, the current mainstream training methods for robust computer vision models are not available. Thus we mainly focus on the safety and reliable issues of computer vision models based on deep neural networks when the real vision data is unavailable， and propose a data-free robustness distillation framework for better reliable applications of CV models.

Submission Number: 7