Robust Ensembles of Neural Networks using Itô Processes

Sumit Kumar Jha, Susmit Jha, Rickard Ewetz, Alvaro Velasquez

Keywords: Robustness, Ito Process, Stochastic

Abstract: Residual neural networks (ResNets) can be modeled as dynamical systems where the evolution of dynamical systems represents the inference in ResNets. We exploit this connection and the theory of stochastic dynamical systems to construct a novel ensemble of Itô processes as a new deep learning representation that is more robust than classical residual networks. An Itô process obtained by solving a suitably-formulated stochastic differential equation derived from a residual network has a probability density function that is not readily perturbed by small changes in the neural network’s inputs. Our robust stochastic Itô ensemble of neural networks achieve an accuracy of 73.91% on the CIFAR-10 dataset against the PGD attack with ε = 2.0 under the L2 norm, while the accuracy of Madry’s robustness toolbox on the same attack is 18.59%. Similarly, our stochastic Itô ensemble of neural networks achieves an accuracy of 79.66% on PGD attack with ε = 16/255 under the L∞ norm, while the accuracy of Madry’s robustness toolbox on the same attack is 18.13%. The Itô ensemble trained on ImageNet achieves an accuracy of 28.53% against PGD attacks under the L∞ norm with ε = 16/255 and accuracy of 65.74% under the L2 norm with ε = 3.0, respectively. This significantly improves state-of-the-art accuracy of 5% and 35.16% for Madry’s robustness tool against the same PGD attacks under the L∞ and L2 norms, respectively. Further, our approach achieves these high robustness values without any explicit adversarial training or a significant loss of accuracy on benign inputs.

One-sentence Summary: We show that ensembles of residual neural networks corresponding to a class of Itô processes are inherently more robust than classical residual networks.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Reviewed Version (pdf): https://openreview.net/references/pdf?id=aRxsBnQ1j