Improving Generalization and Safety of Deep Neural Networks with Masked Anchoring

Vivek Narayanaswamy, Kowshik Thopalli, Rushil Anirudh, Jayaraman J. Thiagarajan

Primary Area: societal considerations including fairness, safety, privacy

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Anomaly Detection, OOD Generalization, ML Safety, Anchoring, Deep Neural Networks

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract: Anchoring is a recent architecture and task-agnostic technique that can produce state-of-the-art epistemic uncertainty estimates, and improve extrapolation capabilities. However, the differences between anchored models and non-anchored variants is not well studied -- as there is little insight into the kinds of functions anchoring induces and how they behave under distribution shifts. In this paper, we analyze and improve anchoring as a training protocol for deep neural networks, evaluating them on important tasks of out of distribution generalization, task adaptation, anomaly detection and calibration. We pinpoint the impact of anchoring on generalization as being inversely related to the sensitivity of the model to the distribution of residuals. We further improve this sensitivity using a new technique called Random Anchor Masking (RAM) that significantly improves the quality of anchored models. We build evidence for the superiority of RAM-training using a range of benchmarks of varying size, using neural networks of varying complexity and scale.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 7985