Adversarially Self-Supervised Pre-training Improves Accuracy and Robustness
Keywords: transfer, pretraining, adversarial, robustness, shift
Abstract: There is growing interest in learning visual representations that work well across distribution shifts as illustrated by the increasing number of ImageNet evaluation sets. In this paper, we reconsider adversarial training, which is generally used as a defense against adversarial shifts, as a way to improve the pre-training of representations for transfer across tasks and natural shifts. In this study we combine adversarial training with different self-supervised pre-training methods such as bootstrap your own latent (BYOL), masked auto-encoding (MAE), and the auxiliary task of rotation prediction (RotNet). We show that the adversarial versions of these self-supervision methods consistently lead to better fine-tuning accuracy both in and out of distribution compared to standard self-supervision, even with nominal/non-adversarial fine-tuning. Furthermore we observe that, to reach best performance with adversarial self-supervised pre-training, (1) the optimal perturbation radius differs among pre-training methods, and (2) that the robust parameters of early layers need to be preserved during fine-tuning to avoid losing the benefits of adversarial pre-training. Finally, we show that there is not a single adversarial self-supervised method that dominates others across all variants, but that adversarial MAE is the best choice for in-distribution variants, and that adversarial BYOL is best for out-of-distribution variants.
Submission Number: 13
Loading