Energy-Based Models Trained with Equilibrium Propagation are Inherently Robust

Siddharth Mansingh, Michal Kucer, Garrett T. Kenyon, Juston Moore, Michael A. Teti

Abstract: Deep neural networks (DNNs) are easily fooled by adversarial perturbations that are impercep-tible to humans. Adversarial training, a process where adversarial examples are added to the training set, is the current state-of-the-art defense against adversarial attacks, but it lowers the model's accuracy on clean inputs, is computationally expensive, and offers less robustness to natural noise. In contrast, energy-based models (EBMs), which were designed for efficient im-plementation in neuromorphic hardware and physical systems, incorporate feedback connections from each layer to the previous layer, yielding a recurrent, deep-attractor architecture which we hypothesize should make them naturally robust. Our work is the first to explore the robustness of EBMs to both natural cor-ruptions and adversarial attacks, which we do using the CIFAR-IO and CIFAR-IOO datasets. We demonstrate that EBMs are more robust than transformers and display comparable robustness to adversarially-trained DNNs on gradient-based (white-box) attacks, query-based (black-box) attacks, and natural perturbations without sacrificing clean accuracy, and without the need for adversarial training or additional training techniques.