Multi-stationary point losses for robust model

Weiwei Gao, Yao Li, Junqi Gao, Zhichang Guo, Dazhi Zhang

Keywords: Robustness, MS loss, Cross-entropy loss, Multi-stationary point losses, Adversarial attack

TL;DR: We propose a familiy of Multi-stationary point losses, which improved robustness.

Abstract: We identify that cross-entropy (CE) loss does not guarantee robust boundary for neural networks. The reason is that CE loss has only one asymptotic stationary point. It stops pushing the boundary forward as long as the sample is correctly classified, which left the boundary right next to the samples. A robust boundary should be kept in the middle of samples from different classes, thus maximizing the margins from the boundary to the samples. In this paper, we propose a family of new losses, called multi-stationary point (MS) losses, which introduce additional stationary points beyond the asymptotic stationary point. We prove that robust boundary can be guaranteed by MS loss without losing much accuracy. With MS loss, bigger perturbations are required to generate adversarial examples. We demonstrate that robustness is improved under a variety of adversarial attacks by applying MS loss. Moreover, robust boundary learned by MS loss also performs well on imbalanced datasets. Finally, we modified other losses into two-stationary-point forms, and improved model robustness is observed.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Deep Learning and representational learning