Towards Parametric Robust Activation Functions in Adversarial Machine Learning

Sheila Alemany, Alberto Luis Dominguez, Ilan Grapel, Niki Pissinou

Keywords: robustness, activation functions, deep learning, adversarial machine learning

TL;DR: This work contributes a parametric activation function that unifies disjointed research efforts for robust activation functions for deep learning.

Abstract: Machine learning's vulnerability to adversarial perturbations has been argued to stem from a learning model's non-local generalization over complex input data. Given the incomplete information in a complex dataset, a learning model captures non-linear patterns between data points with volatility in the loss surface and exploitable areas of low-confidence knowledge. It is the responsibility of activation functions to capture the non-linearity in data and, thus, has inspired disjointed research efforts to create robust activation functions. This work unifies the properties of activation functions that contribute to robust generalization with the generalized gamma distribution function. We show that combining the disjointed characteristics presented in the literature provides more effective robustness than the individual characteristics alone.