Zipper: Decoupling the tradeoff Between Robustness and Accuracy

Hongjun Wang, Yisen Wang

Keywords: Adversarial Training

TL;DR: We propose a bi-expert framework where we simultaneously train base-learners with distribution-aware strategies so that it can obtain both satisfying clean accuracy and robustenss

Abstract: Deep neural networks obtained by standard training have been constantly plagued by adversarial examples. Although adversarial training demonstrates its capability to defend against adversarial examples, unfortunately, training robust classifiers leads to an inevitable drop in the natural generalization when performing adversarial training. To address the issues, we decouple the standard generalization and the robust generalization from joint training and formulate different training strategies for each one. Specifically, instead of minimizing a global loss on the expectation over these two generalization errors, we propose a bi-expert framework called \emph{Zipper} where we simultaneously train base learners with distribution-aware strategies so that they can specialize in their own fields. The parameters of base learners are collected and combined to form a global learner at intervals during the training process, which is then distributed to base learners as initialized parameters for continued training. Theoretically, we show that the risks of Zipper will get lower once the base learners are well trained. Extensive experiments verify the applicability of Zipper to satisfying high clean accuracy in the natural setting while keeping considerably robust to the adversarial setting, compared to relevant techniques.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Social Aspects of Machine Learning (eg, AI safety, fairness, privacy, interpretability, human-AI interaction, ethics)