On 3D Reconstruction Pre-training to Improve Adversarial Robustness
Abstract: Ensuring robustness of image classifiers against worst-case adversarial perturbations has been challenging. A promising idea is to build models that use robust features, instead of non-robust features that nevertheless generalize to test sets. One of the most effective methods for inducing such robust features is a type of data augmentation that uses adversarial examples during training. Here, inspired by studies of human vision, we explore a synthesis of this approach by leveraging a causal property underlying image formation: the 3D geometry of objects and how it projects to images. We combine adversarial training with a weight initialization that encodes prior knowledge about 3D objects, which is achieved via 3D reconstruction pre-training. We evaluate our approach using two different datasets and compare it to alternative non-3D pre-training protocols. To systematically explore the effect of 3D pre-training, we introduce a novel dataset called Geon3D, which consists of simple shapes that nevertheless capture variation in multiple distinct dimensions of geometry. We find that while 3D reconstruction pre-training does not improve robustness for the simplest dataset setting we consider (Geon3D on a clean background), it improves upon adversarial training in more realistic (Geon3D with textured background) and challenging (spurious correlations between shape and background textures) dataset conditions, as well as on a dataset with a more complex distribution of shapes (ShapeNet). Furthermore, we show that the benefit of using 3D-based pre-training outperforms 2D-based pre-training on ShapeNet. We hope that these results encourage further investigation of the benefits of 3D vision for adversarial robustness.
Submission Length: Regular submission (no more than 12 pages of main content)
Assigned Action Editor: ~Charles_Xu1
Submission Number: 710
Loading