Adversarial Rademacher Complexity of Deep Neural Networks

Jiancong Xiao, Yanbo Fan, Ruoyu Sun, Zhi-Quan Luo

Keywords: Adversarial Robustness, Generalization, Rademacher complexity

Abstract: Deep neural networks are vulnerable to adversarial attacks. Adversarial training is one of the most effective algorithms to increase the model's robustness. However, the trained models cannot generalize well to the adversarial examples on the test set. In this paper, we study the generalization of adversarial training through the lens of adversarial Rademacher complexity. Current analysis of adversarial Rademacher complexity is up to two-layer neural networks. In adversarial settings, one major difficulty of generalizing these results to deep neural networks is that we cannot peel off the layer as the classical analysis for standard training. We provide a method to overcome this issue and provide upper bounds of adversarial Rademacher complexity of deep neural networks. Similar to the existing bounds of standard Rademacher complexity of neural nets, our bound also includes the product of weight norms. We provide experiments to show that the adversarially trained weight norms are larger than the standard trained weight norms, thus providing an explanation for the bad generalization performance of adversarial training.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2211.14966/code)