Hessian-Aware Training for Enhancing Model Resilience for In-Memory Computing

Tahmid Hasan Pranto, Seijoon Kim, Lizhong Chen, Sanghyun Hong

Keywords: DNN Resilience, Parameter corruptions, Hessian-aware training

TL;DR: We propose hessian-aware training to enhance the resilience of deep neural networks to parameter-level variations.

Abstract: Deep neural networks are not resilient to bitwise errors in their parameters: even a single-bit error in their memory representation can lead to significant performance degradation. This susceptibility poses great challenges in deploying models on emerging computing platforms, such as in-memory computing devices, where frequent bitwise errors occur. Most prior work addresses this issue with hardware or system-level approaches, such as additional hardware components for checking a model’s integrity at runtime. However, these methods have not been widely deployed since they necessitate substantial infrastructure-wide modifications. In this paper, we study a new approach to address this challenge: we present a novel training method aimed at enhancing a model’s inherent resilience to parameter errors. We define a model-sensitivity metric to measure this resilience and propose a training algorithm with an objective of minimizing the sensitivity. Models trained with our method demonstrate increased resilience to bitwise errors in parameters, particularly with a 50% reduction in the number of bits in the model parameter space whose flipping leads to a 90–100% accuracy drop. Our method also aids in extreme model compression, such as lower bit-width quantization or pruning ∼70% of parameters, with reduced performance loss. Moreover, our method is compatible with existing strategies to mitigate this susceptibility.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 7765