Certifying Robust Generalization with Diverging Spanned Latent Space
Abstract: Robust generalization (RG), concerning how deep neural networks could perform over adversarial examples generated from unseen dataset, has emerged as an active research topic. Albeit its crucial importance, most previous studies lack a well-founded theoretical analysis and certified error bounds. In this paper, we make a novel attempt to theoretically and empirically study how we could attain a better RG by learning discriminative representation, where the inconsistency of the inter-sample similarity matrix between clean and adversarial examples should be reduced. Our theoretical investigation discloses that introducing this inconsistency as a regularization term, named Gram matrix difference (GMD), will lead to tighter upper error bound and certify a better RG. Meanwhile, we demonstrate that previous efforts to reduce inter-class similarity and increase intra-class similarity among adversarial examples for enhanced adversarial robustness are approximate optimizations of our GMD approach. Furthermore, to avoid the vast optimization complexity introduced by the similarity matrix, we propose to optimize GMD by building a diverging spanned latent space for adversarial examples. On the algorithmic side, this regularization term is implemented as a novel adversarial training (AT) method --- Subspace Diverging (SD) --- to expand the volume difference between the whole latent space's linear span and subspaces' linear spans. Extensive experiments show that the proposed method can improve advanced AT methods and work remarkably well in various datasets including CIFAR-10, CIFAR-100, SVHN, and Tiny-ImageNet.
Submission Length: Long submission (more than 12 pages of main content)
Assigned Action Editor: ~Charles_Xu1
Submission Number: 3018
Loading