Task-Agnostic Unsupervised Robust Representation Learning

A. Tuan Nguyen, Philip Torr, Ser-Nam Lim

Keywords: unsupervised robustness, transferable adversarial robustness

Abstract: It has been reported that deep learning models are extremely vulnerable to small but intentionally chosen perturbations of their input. In particular, a deep network, despite its near-optimal accuracy on the clean images, often mis-classifies an image with a worst-case but humanly imperceptible perturbation (so-called adversarial examples). To tackle this problem, a great amount of research has been done to study the training procedure of a network to improve its robustness. However, most of the research so far has focused on the case of supervised learning. With the increasing popularity of self-supervised learning methods, it is also important to study and improve the robustness of their resulting representation on downstream tasks. In this paper, we study the problem of robust representation learning with unlabeled data in a task-agnostic manner. Specifically, we first derive an upper bound on the adversarial loss of a prediction model (which is based on the learned representation) on any downstream task, using its loss on the clean data and a robustness regularizer. Importantly, the regularizer is task-independent, thus we propose to minimize it directly during the representation learning phase to make the downstream prediction model more robust. Extensive experiments show that our method results in a robust model for downstream tasks without any supervised adversarial training, and achieves preferable adversarial performance compared to relevant baselines.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Unsupervised and Self-supervised learning

TL;DR: We propose a method to learn robust representations without any labels or adversarial fine-tuning in downstream tasks, based on a theoretically grounded unsupervised robustness regularizer.