Efficient Robust Training via Backward Smoothing

Jinghui Chen, Yu Cheng, Zhe Gan, Quanquan Gu, Jingjing Liu

Keywords: Efficient Robust Training, Backward Smoothing, Robustness

Abstract: Adversarial training is so far the most effective strategy in defending against adversarial examples. However, it suffers from high computational cost due to the iterative adversarial attacks in each training step. Recent studies show that it is possible to achieve Fast Adversarial Training by performing a single-step attack with random initialization. Yet, it remains a mystery why random initialization helps. Besides, such an approach still lags behind state-of-the-art adversarial training algorithms on both stability and model robustness. In this work, we develop a new understanding towards Fast Adversarial Training, by viewing random initialization as performing randomized smoothing for better optimization of the inner maximization problem. From this perspective, we show that the smoothing effect by random initialization is not sufficient under the adversarial perturbation constraint. A new initialization strategy, \emph{backward smoothing}, is proposed to address this issue and significantly improves both stability and model robustness over single-step robust training methods. Experiments on multiple benchmarks demonstrate that our method achieves similar model robustness as the original TRADES method, while using much less training time (~3x improvement with the same training schedule).

One-sentence Summary: We propose a new principle towards understanding Fast Adversarial Training, and a new initialization strategy that significantly improves both stability and model robustness over the single-step robust training methods.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:2010.01278/code)

Reviewed Version (pdf): https://openreview.net/references/pdf?id=aMIRgsXyhS