Hessian-aware Training for Enhancing DNN Resilience to Bitwise Corruptions in Their Parameters

TMLR Paper5889 Authors

Abstract:

Deep neural networks are not resilient to parameter corruptions: even a single-bitwise error in their parameters in memory can cause an accuracy drop of over 10%, and in the worst-cases, up to 99%. This susceptibility poses great challenges in deploying models on computing platforms, where adversaries can induce random/targeted bit-flips, e.g., through software-induced fault attacks like Rowhammer. Most prior work addresses this issue with hardware or system-level approaches, such as integrating additional hardware components to verify a model’s integrity at inference. However, these methods have not been widely deployed as they require infrastructure or platform-wide modifications.

In this paper, we propose a new approach to addressing this issue: training models to be more resilient to bitwise corruptions to their parameters. Our approach, Hessian-aware training, promotes models to learn flatter loss surfaces. We show that existing training methods designed to improve generalization (e.g., through sharpness-aware minimization) do not enhance resilience to parameter corruptions. In contrast, models trained with our method demonstrate improved resilience to parameter corruptions, particularly with a 20–50% reduction in the number of bits whose individual flipping leads to a 90–100% accuracy drop. We also characterize the factors that may influence this increased resilience. Moreover, we show the synergy between ours and existing hardware and system-level defenses.

Submission Type: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Jonathan_Ullman1

Submission Number: 5889