A Study of the Effects of Transfer Learning on Adversarial Robustness

Pratik Vaishnavi, Kevin Eykholt, Amir Rahmati

Primary Area: transfer learning, meta learning, and lifelong learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: adversarial robustness, adversarial training, certified robustness, randomized smoothing, transfer learning

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract:

The security and robustness of AI systems are critical in real-world deployments. While prior works have developed methods to train robust networks, these works implicitly assume that sufficient labeled data for robust training is present. However, in deployment scenarios with insufficient training data, robust networks cannot be trained using existing techniques. In such low-data regimes, non-robust training methods traditionally rely on *transfer learning*. First, a network is pre-trained on a large, possibly labeled dataset and then fine-tuned for a new task using the smaller set of training samples.
The effectiveness of transfer learning with respect to adversarial robustness, though, is not well-studied. It is unclear if transfer learning can improve adversarial performance in low-data scenarios. In this paper, we perform a broad analysis of the effects of pre-training with respect to empirical and certified adversarial robustness. Using both supervised and self-supervised pre-training methods across a range of downstream tasks, we identify the circumstances necessary to train robust models on small-scale datasets. Our work also represents the first successful demonstration of training networks with high certified robustness for small-scale datasets.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 2789