Revisiting Fast Adversarial Training

Xiaojun Jia, YueFeng Chen, Xiaofeng Mao, Jindong Gu, Ranjie Duan, Rong Zhang, Hui Xue', Xiaochun Cao

Keywords: adversarial training，model robustness, adversarial examples

Abstract: Fast Adversarial Training (FAT) not only improves the model robustness but also reduces the training cost of standard adversarial training. However, FAT often suffers from Catastrophic Overfitting (CO), which results in poor robustness performance. CO describes the phenomenon that model robust accuracy can decrease dramatically and suddenly during the training of FAT. Many effective techniques have been developed to prevent CO and improve the model robustness from different perspectives. However, these techniques adopt inconsistent training settings and require different training costs, i.e, training time and memory costs, resulting in an unfair comparison. In this paper, we first conduct a comprehensive study of more than 10 FAT methods in terms of adversarial robustness and training costs. We revisit the effectiveness and efficiency of FAT techniques in preventing CO from the perspective of model local nonlinearity and propose an effective Lipschitz regularization method for FAT. Furthermore, we explore the effect of data augmentation and weight averaging in FAT and propose a simple yet effective auto weight averaging method to improve robustness further. By assembling these techniques, we propose a FGSM-based fast adversarial training method equipped with Lipschitz regularization and Auto Weight averaging, abbreviated as FGSM-LAW. Experimental evaluations on four benchmark databases demonstrate the superiority of the proposed method.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Social Aspects of Machine Learning (eg, AI safety, fairness, privacy, interpretability, human-AI interaction, ethics)