Robustness through Random Activation: Adversarial Training with Bernoulli Rectified Linear Units

JaeHyeon Kim, Jay Hoon Jung

Keywords: Adversarial training, Adversarial attack, Activation function

TL;DR: The robustness of the model can be further improved by using our proposed activation function in adversarial training.

Abstract:

Despite their considerable achievements across a range of domains, deep learning models have been demonstrated to be susceptible to adversarial attacks.
In order to mitigate this vulnerability, adversarial training has become a prevalent defense strategy.
In this context, we propose Bernoulli Rectified Linear Units (BReLU), an activation function designed to further enhance the effectiveness of adversarial training. In contrast to conventional activation functions, BReLU modulates activation probabilities in accordance with input values, thereby introducing input-dependent randomness into the model.
The experimental results demonstrate that the incorporation of BReLU into adversarial training significantly enhances the robustness of the model against adversarial attacks. Specifically, on the CIFAR-10 dataset using the ResNet-18 model, BReLU improved robustness by 15\% under FGSM, by 8\% under PGD-20, and by 54\% under the CW attack compared to ReLU.
Our findings indicate that BReLU represents a promising addition to adversarial training techniques for strengthening deep learning models against adversarial threats.

Primary Area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 9576