Sparta: Spatially Attentive and Adversarially Robust Activations

Qing Guo, Felix Juefei-Xu, Changqing Zhou, Lei Ma, Xiaofei Xie, Wei Feng, Yang Liu

Keywords: adversarial training, activation function, spatially attentive activation

Abstract: Adversarial training has been demonstrated to be useful for improving the robustness of deep neural networks (DNNs). However, the impacts of basic network components (e.g., ReLU, the widely used activation function for DNNs) to adversarial training effectiveness received less attention and has not been comprehensively investigated so far. To fill this gap, in this work, we argue that the spatially-shared and input-independent activating properties of the ReLU make the DNNs under both standard training and adversarial training less robust to white-box adversarial attacks. To address such challenges, we design a novel activation function, i.e., Sparta: Spatially Attentive and Adversarially Robust Activation, which enables DNNs to achieve higher robustness (i.e., lower error rate on adversarial examples) and accuracy (i.e., lower error rate on clean examples) than the DNNs based on the state-of-the-art activation functions. We further investigate the relationships between our Sparta and the state-of-the-art search-based activation function, i.e., Swish, and feature denoising method, providing insights about the advantages of our method. Moreover, comprehensive evaluations have demonstrated two important properties of our method: First, superior transferability across DNNs. Our adversarially trained Sparta function for one DNN (e.g., ResNet-18) can be fixed to train another adversarially robust DNN (e.g., ResNet-34), achieving higher robustness than the one using vanilla ReLU as activation. Second, superior transferability across datasets. The Sparta function trained on one dataset (e.g., CIFAR-10) can be employed to train adversarially robust DNNs on another dataset (e.g., SVHN) and helps achieve higher robustness than DNNs with vanilla ReLU as activation. These properties have highlighted the flexibility and versatility of Sparta. Accompanying code is also submitted.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: We propose the spatially attentive and adversarially robust activation function for more effective adversarial training.

Reviewed Version (pdf): https://openreview.net/references/pdf?id=VKw_K412Eq