Improving model robustness against noise with safe haven activations

Zhanglu Yan, Dogukan Yigit Polat, Shida Wang, Kaiwen Tang, Weng-Fai Wong

Keywords: Quantized Neural Networks, Noise Robustness

Abstract: Quantized neural networks (QNNs) are often used in edge AI because they reduce memory and computational demands. In practical applications such as control systems, medical imaging, and robotics, controlling input noise is crucial for enhancing system robustness. Thus, improving the noise resilience of QNNs is an important challenge in achieving effective edge AI applications. In this paper, we investigate the impact of input noise on QNN performance and propose the safe haven activation quantization (SHAQ) method. This approach leverages the characteristics of the quantization function to constrain outputs before quantization within a more noise-resilient 'safe' range, effectively reducing the impact of noise across quantized layers. Our methods achieve state-of-the-art, 73.11\% accuracy with 2-bit activations under the fast gradient sign method (FGSM) adversarial attacks with an epsilon of 8/255 on the CIFAR-10 dataset. Furthermore, we extend our methods into a plug-and-play solution we call quantized helmet (QH), comprising a series of quantized layers that can be integrated into any unquantized neural network to enhance its noise robustness. Our experimental code and analysis are open-source and publicly accessible.

Primary Area: optimization

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 7635