Gradient norm as a powerful proxy to out-of-distribution error estimation

RENCHUNZI XIE, Ambroise Odonnat, Vasilii Feofanov, Ievgen Redko, Jianfeng Zhang, Bo An

Primary Area: transfer learning, meta learning, and lifelong learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: out-of-distribution error estimation, machine learning, deep learning

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We use the norm of classification-layer gradients, backpropagated from the cross entropy loss with only one gradient step over OOD data, to formulate an estimation score correlating with the expected OOD error.

Abstract: Estimating out-of-distribution (OOD) error without access to the ground-truth test labels is a highly challenging, yet extremely important problem in the safe deployment of machine learning algorithms. Current works rely on the information from either the outputs or the extracted features to formulate an estimation score correlating with the expected OOD error. In this paper, we investigate--both empirically and theoretically--how the information provided by the gradients can be predictive of the OOD error. Specifically, we use the norm of classification-layer gradients, backpropagated from the cross-entropy loss with only one gradient step over OOD data. Our key idea is that the model should be adjusted with a higher magnitude of gradients when it does not generalize to the OOD dataset. We provide theoretical insights highlighting the main ingredients of such an approach ensuring its empirical success. Extensive experiments conducted on diverse distribution shifts and model structures demonstrate that our method outperforms state-of-the-art algorithms significantly.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 4544